AD- A 142  449 


•.search  Note  84-78 


EARLY  TRAINING  ESTIMATION  SYSTEM  (ETES) 
FINAL  REPORT 


Lawrence  H.  O'Brien 
Dynamics  Research  Corporation 


Charles  Jorgensen,  Contracting  Officer's  Representative 


Submitted  by 

Michael  H.  Strub,  Chief 
ARI  FIELD  UNIT  AT  FORT  BLISS,  TEXAS 


Jerrold  M.  Levine,  Director 
SYSTEMS  RESEARCH  LABORATORY 


lie 


U.  S.  Army 

Research  Institute  for  the  Behavioral  and  Social  Sciences 


June  1984 


Approved  for  public  rtleaee:  diitribution  unlimited. 


NAT  O^L  TECHNICAL 

information  service 


Jhit  report,  at  tubmittad  by  ttv  contractor,  hat  been  claared  for  rateate  to  Detente  Technical  Information  Center 
(OTICI  to  comply  with  reQuIatr  ry  requirementt.  It  hat  been  given  no  primary  dittribution  other  than  to  OTIC 
and  wilt  be  available  only  through  DTIC  or  other  reference  tervicet  tuch  at  the  National  Technical  Information 
Service  (NTIS).  The  viewt.  epiniont,  aitd/or  findingt  contaiited  in  thit  report  are  thote  of  the  author(t)  and 
thould  not  be  conttrueu  at  an  official  Department  of  the  Army  petition,  policy,  or  decition,  unlett  to  detignated 
by  other  official  documentatioi. 


84 


2  6 


DISCLAIHEt  None 


TfflS  DOCUMENT  IS  BEST 
QUALITY  AVAILABLE.  THE  COPY 
FURNISHED  TO  DTIC  CONTAINED 
A  SIGNIFICANT  NUMBER  OF 
PAGES  WHICH  DO  NOT 
REPRODUCE  LEGIBLY. 


REPORT  DOCUMENTATION  PAGE 


1.  REPORT. NUMBER 


Research  Note  84-78 


4.  (and  Subttttm) 


EARLY  TRAINING  ESTIMATION  SYSTEM;  FINAL  REPORT 


READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


3.  RECIPIENT'S  CATALOG  HUMBER 


S.  TYPE  OF  REPORT  A  PERIOD  COVERED 


FINAL 


6.  PERFORMING  ORG.  REPORT  NUMBER 


7.  AUTHORf*^ 

Lawrence  H.  O'Brien 


8.  CONTRACT  OR  grant  NOMBERf#; 

MDA-903-80-C-0525 


9.  performing  organization  name  and  address 
DYNAMICS  RESEARCH  CORPORATION 
60  Concord  Street 
Wilmington,  Massachusetts  01887 


11.  CONTROLLING  OFFICE  NAME  AND  ADDRESS 

Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  Virginia  22333 


U.  monitoring  AGENCY  NAME  A  AOCRESSC//  dltlarmnt  from  Controlling  Olllco) 

US  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences 
Fort  Bliss  Field  Unit,  P.0,  Box  6057 
Fort  Bliss.  TX  79916 _ 


*6.  DISTRIBUTION  STATEMENT  (oi  this  Raport) 

Approved  for  public  release;  distribution  unlimited 


17,  OlSTRteuTION  STATEMENT  (a(  mbatr^ct  enlarad  in  Btoek  20,  H  dittatant  from  Rapori) 


10.  PROGRAM  element,  PROJECT,  TASK 
AREA  ft  WORK  UNIT  NUMBERS 


2Q162722A791 


11.  report  date 
June  1984 


13.  NUMBER  OF  PAGES 

237 


IS.  SECURITY  CLASS,  (al  Ihim  rapon) 

UNCLASSIFIED 


IS..  DECLASSIFICATION/ downgrading 
schedule 


18.  supplementary  notes 


Charles  Jorgensen, Contracting  Officers  Representative 


19.  KEY  WORDS  (Contlnua  on  rmvarma  aida  U  nacaaamiy  and  Identify  by  block  numbar) 


Training 

Instructional  System  Development 
Data  Base  Management 
Task  Analysis 


Training  Estimation 
Simulation 

Training  Effectiveness 
Training  Effectiveness  Analysis 
Front  EniLAnalysis 


20,  At:ST7;  ACT  fCstrtinu#  eta  rararaa  at-^  tf  na^ra^eary  aod.  tdertilfy  by  block  nunbarj 

This  report  describes  the  research  and  development  activities  conducted  under 
the  Early  Training  Estimation  System  (ETES)  development  project.  The  Early 
Training  Estimation  System  (ETES)  is  an  integrated  set  of  procedures  and 
automated  tools  for  estimating  training  requirements  during  the  earliest 
phases  of  the  weapon  system  acquisition  process.  The  ETES  has  three  major 
components;  a  System  Description  Technology  (SDT) ,  Early  Training  Estimation 
Aids  and  Procedures  (TEAP) ,  and  Evaluative  Technology.  The  SDT  is  a  data 
base  management  system  for  storing  and  tracking  task  and  training-related _ 


I3AN73N73  EOmON  OF  *  NOV  65  IS  OBSOLETE  I IWPI  fl  f  P  T  P  H 


EDITION  OF  »  NOV  65  IS  OBSOLETE 


UNCLASSIFIED 

security  CLASSIFICATION  OF  THIS  PAGE  (Iffran  Daia  Enlaratl) 


_ Unclassified _ 

security  classification  of  this  P AGE(T>h«n  Dmta  Entara^ 


data.  The  data  in  the  SOT  is  used  in  the  TEAP  to  estimate  training  require¬ 
ments  for  a  new  system.  These  training  requirements  include  estimates  of 
task  requirements,  course  requirements,  and  resource  requirements  as  well  as 
estimates  of  training  costs,  training  efficiency,  and  training  effectiveness. 
In  the  Evaluative  Technology,  the  integrated  impacts  of  training  requirements 
are  assessed,  training  alternatives  are  evaluated,,  tradeoff  and  sensitivity 
analyses  of  key  parameters  are  conducted,  and  the  relationships  between  ETES 
outputs  and  key  Army  acquisition  documents  and  processes  are  specified. 

This  report  provides  an  overview  of  the  components  of  ETES,  describes  the 
research  conducted  under  each  of  the  five  ETES  study  tasks;  and  outlines 
future  directions  for  improving  ETES. 

The  final  report  and  Appendixes  are  published  as  separate  volumes  as  follows: 
Final  Report:  ARI  Research  Note  84-78  (includes  Appendixes  A  through  E) 
Appendix  F,  User's  Guide:  ARI  Research  Note  84-79 

Appendix  G,  User's  Guide,  System  Description  Technology:  ARI  Research  Note 
84-80 

Appendix  H,  User's  Guide,  Media  Selection  Program:  ARI  Research  Note  84-81 

Appendix  I,  User's  Guide,  Automated  Resource  and  Cost  Estimation  Technique: 
ARI  Research  Note  84-82 

Appendix  J,  User's  Guide,  Automated  Planning  and  Scheduling  Technique  for 
Individual  and  Collective  Training  Plan:  ARI  Research  Note  84-83 


UNCLASSIFIED 


security  CU  ASSI  P  iC  a  T  lOs  OF  This  P  AGEl'^'T-wn  Dnia  Entermd) 


TABLE  OF  CONTENTS 


Section  Page 

1.0  Introduction  and  Summary  1-1 

1.1  Objectives  1-1 

1.2  Organization  1-1 

1.3  Need  for  Early  Training  Estimation  1-3 

1.3.1  Requirements  for  An  Early  Training 

Estimation  System  1-6 

1.3.2  Intended  Users  of  ETES  1-7 

1.4  ETES  Study  Tasks  1-8 

1.4.1  Original  ETES  Concept  1-8 

1.4.2  Task  1:  Assess  Existing  Concept 

Development  Procedures  1-11 

1.4.3  Task  2:  Develop  Method  for  System 

Concept  Description  1-12 

1.4.4  Task  3;  Develop  Training  Estimation 

Aids/Procedures  1-13 

1.4.5  Task  4:  Develop  Evaluative  Technology  1-14 

1.4.6  Task  5:  Develop  User's  Guides  1-15 

1.5  Current  ETES  Components  1-15 

1.5.1  The  SDT  1-17 

1.5.2  Training  Estimation  Aids/Procedures  1-26 

1.5.3  Evaluative  Technology  1-3  7 

1.6  Relationships  Between  ETES  and  Other  ARI 

Products  1-4  2 

2.0  Task  1;  Assess  Existing  Concept  Development 

Procedures  2-1 

2.1  Assess  LCSMM  Interfaces  2-1 

2.1.1  OICTP/ICTP  2-2 

2.1.2  CTEA  2-6 

2.1.3  Identification  of  LCSMM  Reports  2-10 

2.2  Interview  Acquisition  Participants  2-17 

2.3  Review  Psychological  Aspects  of  Design  Process  2-2  1 


TABLE  OF  CONTENTS  (continued) 


Section  Page 

3.0  Task  2:  Develop  Method  for  System  Concept 

Description  3-1 

3.1  Review  Requirements  Analyses  Tools  3-2 

3.2  Review  Data  Base  Tools  3-6 

3.2.1  Overview  of  DBMS  Concepts  3-7 

3.2.2  Process  and  Results  of  Reviewing  Data 

base  Management  Techniques  3-11 

3.3  Review  Research  Related  To  Human  Computer 

Interaction  3-3  7 

3.3.1  SDT  Requirements  Related  To  Human 

Computer  Interaction  3-38 

3.3.2  Human-Data  Base  Interaction  Literature 

Review  3-3  9 

3.3.3  Summary  of  Implications  for  SDT  3-53 

3.4  Develop  SDT  3-5  5 

3.4.1  Identify  SDT  Requirements  3-55 

3.4.2  Identify  Data  Elements  3-5  5 

3.4.3  Identify  Data  Structure  3-57 

3.4.4  Identify  Frames  and  Program  Logic  3-60 

3.4.5  Develop  Software  3-6  0 

3.4.6  Test/Revise  Software  3-60 

3.5  Apply  SDT  to  Army  Weapon  System  3-6  2 

4.0  Task  3:  Develop  Training  Estimation  Aids 

and  Procedures  4-1 

4.1  Identify  Functional  Requiranents  4-1 

4.2  Identify  Functions  and  Tools  To  Be  Included 

In  TEAP  4-8 

4.3  Select  Tools  To  Be  Automated  4-12 

4.4  Develop  TEAP  4-13 


5.0  Task  4;  Develop  Evaluative  Technology 

5.1  Review /Examine  Simulation  Techniques 

5.2  Develop  Evaluative  Technology 


5-1 

5-1 

5-3 


TABLE  OF  CONTENTS  (continued) 


Section 


Page. 


5.2.1  Identify  Functional  Requirements 

5.2.2  Identify  Functions  and  Tools  To  Be 
Included  In  Evaluative  Technology 

5.2.3  Select  Tools  To  Be  Automated 

5.2.4  Develop  Evaluative  Technology 

6.0  Task  5:  Develop  User's  Guide 

7.0  Potential  Improvements 

7.1  SDT 

7.2  Training  Estimation  Aids  and  Procedures 

7.3  Evaluative  Technology 

7.4  General  Areas  of  Improvement 

References 


5-5 

5-9 

5-10 


Appendix  A;  Overview  of  Media  Selection  Program 

Appendix  B:  Overview  of  Automated  Resource  and 
Cost  Estimation  Technique 

Appendix  C:  Overview  of  Automated  Training 

Planning  and  Scheduling  Technique 

Appendix  D;  Description  of  ARI  Projects  Related 
to  ETES 

Appendix  E:  Documents  Reviewed  During  Examination 
of  LCSMM  Process 


V 


LIST  OF  FIGURES 


Figure 

1-1  System  Definition  As  A  Function  of  Acquisition 
Phase 

1-2  ETES  Components 
1-3  ETES  Architecture 

1-4  System  Development  Process  for  SDT 
1-5  Training  Estimation  Aids/Procedures 
1-6  Relationship  Between  ETES  and  ISD 
1-7  Evaluative  Technology 

3-1  An  Architecture  for  a  Data  Base  System 
3-2  Overview  of  Processes  in  SDT  Development 
3-3  Relationships  Among  SDT  Entities 

3- 4  Example  of  SDT  Menu 

4- 1  Procedures  for  Developing  Training  Estimation 

Aids/Procedures 

4- 2  Procedures  for  Developing  Automated  Training 

Estimation  Aids/Procedures 

5- 1  Procedures  for  Developing  Evaluative  Technology 

A-1  Overview  of  Media  Selection  Program 
B-1  Overview  of  RCET  Procedures 
C-1  Overview  of  Procedures  for  Using  Automated 
Planning  and  Scheduling  Techniques 


LIST  OF  TABLES 


Table 


1-1  ETES  Study  Tasks 

1-2  Comparison  of  Initial  and  Current  ETES  Concept 

1-3  SDT  Data  Elements 

1-4  Major  Army  Acquisition  Processes  and  Documents 
Related  to  Early  Training  Estimation 

1- 5  Relationships  Between  Projects  in  ARI  Thrust 

2- 1  LCSMM  Documents  and  Processes  Related  to  Early 

Training  Estimation 

2-2  Relationship  Between  OICTP  and  ETES  Functions 

2-3  Relationship  Between  CTEA  and  ETES  Functions 

2-4  Summary  of  Standardized  Reports  Related  to 
Early  Training  Estimation 

2-5  Organizations  Interviewed  During  Task  1 

2- 6  Potential  Problem  Solving  Aids 

3- 1  Requirements  Methodologies  and  Results  of 

Preliminary  Assessment 

3-2  Characteristics  of  Commercially  Available 
Mainframe/Minicomputer  Based  DBMSs 

3-3  SDT  Hardware  Configuration 

3-4  Past  Efforts  At  Human  Resource  Data  Base 
Development 

3-5  ■  Overview  of  LSAR  and  Its  Major  Weaknesses 

3-6  Limitations  of  UDB 

3-7  Data  Elements  Contained  in  CDB 

3-8  SAT  Data  Elements  and  Limitations 

3-9  NEPDISS  Data  Limitations 

3-10  Martin's  Dialogue  Types  and  Their  Applicability 
to  SDT 

3-11  Martin  Categorization  of  Input-Output  Devices 
And  Their  Applicability  to  SDT 
3-12  Ramsey  and  Atwood's  Scheme  for  Classifying 
Human  Computer  Interaction  Information 
3-13  Ramsey  and  Atwood's  Scheme  for  Classifying 
Dialogue  Types 

3-14  Smith's  Scheme  for  Classifying  Human-Computer 
Interaction  Information 

3-15  Smith's  Scheme  for  Classifying  Dialogue  Types 
3-16  Sidorsky's  and  Parrish's  Scheme  For  Classifying 
Human-Computer  Interaction  Information 
3-17  Sidorsky's  and  Parrish's  Scheme  for 
Highlighting  Information 
3-18  Example  of  SDT  Data  Dictionary  Page 


LIST  OF  TABLES  (continued) 


Page 

Functional  Requirements  for  Training 

Aids/Procedures  4-3 

Focus  of  ETES  Training  Estimation 

Aids/Procedures  4-11 

Simulation  Techniques  Reviewed  During  Task  4  5-2 

Functional  Requirements  for  Evaluative 

Technology  5-7 

ETES  Documentation  6-2 

Potential  Areas  of  Improvement  7-2 

Possible  Assignment  Strategies  A-6 

Example  Cost  Elements  B-3 


SECTION  1 


INTRODUCTION  AND  SUMMARY 


1.1  OBJECTIVES 

This  report  summarizes  the  research  and  development 
activities  conducted  during  the  Early  Training  Estimation 
System  (ETES)  project.  The  ETES  project  was  a  three  year 
effort  sponsored  by  the  Army  Research  Institute.  The 
objective  of  ETES  was  to  produce  an  integrated  set  of 
procedures  and  automated  tools  for  estimating  training 
requirements  for  emerging  Army  weapons  systems  during  the 
earliest  phases  of  the  acquisition  process  (Mission  Area 
Analysis,  Concept  Exploration,  and  Demonstration  and 
Validation ) . 

1.2  ORGANIZATION 

The  remainder  of  this  section  describes  (a)  the  problem 
areas  that  ETES  was  designed  to  address,  (b)  the  intended 
users  of  ETES,  (c)  the  ETES  study  tasks,  (d)  the  components 
of  ETES,  and  (e)  the  relationship  between  ETES  and  other  ARI 
projects  designed  to  assess  manpower,  personnel,  and 
training  requirements  for  developing  Army  systems. 

Sections  2  thru  6  provide  a  detailed  description  of  the 
research  conducted  under  each  of  the  five  ETES  development 
tasks . 

Section  7  describes  potential  areas  for  improving  the 
current  ETES.  Appendices  A,  B,  and  C  describe  the 
algorithms  and  software  in  the  three  automated  ETES  training 


estimation  tools  —  Media  Selection  Program,  Resource  and 
Cost  Estimation  Technique  and  Automated  Training,  Planning, 
and  Scheduling  Technique.  Appendix  D  describes  other  ARI 
projects  related  to  ETES.  Appendix  E  lists  the  Life  Cycle 
System  Management  Model  documents  reviewed  during  the  ETES 
study. 

Detailed  descriptions  of  ETES  automated  and  procedural  tools 
are  available  in  the  following  documents: 


o  ETES  Procedural  Guide  (approximately  300  pages, 

o  User's  Guide:  ETES  System  Description  Technology 
(approximately  300  pages) 

o  User's  Guide:  ETES  Media  Selection  Program 

(approximately  200  pages) 

o  User's  Guide:  ETES  Resource  and  Cost  Estimation 

Technique  (approximately  100  pages) 

o  User's  Guide:  ETES  Automated  Training,  Planning 

and  Scheduling  Technique  (approximately  100  pages) 


Additional  descriptions  of  ETES  tools  are  provided  in  the 
following  technical  papers: 

Jorgensen,  C. ,  and  O'Brien,  L.  The  Early  Training  Estimation 
System:  An  automated  training  needs  assessment  technique. 


presented  at  the  Second  Annual  Conference  on  Microcomputers 
in  Education,  Washington,  June  1982. 


Boylston,  D.  An  automated  decision  aid  for  the  assignment 
of  tasks  to  training  media  in  early  training  estimation. 
Paper  to  be  presented  at  the  51st  Symposium  of  the  Military 
Operations  Research  Society,  September  27-29,  1983. 

O'Brien,  L.  Early  Training  Estimation  System  (ETES). 
Proceedings  of  the  TRADOC  Developments  Institute  Chiefs  of 
Analysis  Seminar,  October  21-13,  1981. 


1.3  NEED  FOR  EARLY  TRAINING  ESTIMATION 

The  Early  Training  Estimation  System  provides  a  capability 
for  systematically  estimating  training  requirements  for 
developing  Army  weapon  systems  during  the  earliest  phases  of 
the  acquisition  process  (Mission  Area  Analysts,  Concept 
Exploration  -  Phase  I,  and  Demonstration  and  Validation  - 
Phase  II).  These  estimates  of  training  requirements  include 
specification  of  the  system's  tas)<  requirements,  training 
course  requirements,  resource  requirements,  and  estimates  of 
training  cost  and  "effectiveness."^  There  are  two  major 
reasons  why  such  early  estimates  of  training  requirements 
are  needed. 


ETES  only  provides  gross  high  level  estimates  of 
training  effectiveness.  These  estimates  are  only 

appropriate  during  the  earliest  phases  of  the  acquisition 
process . 


First,  by  developing  earlier  and  more  accurate  estimates  of 
training  requirements,  the  training  planning  process  can 
begin  earlier.  As  a  result,  the  training  products 

associated  with  a  system,  many  of  which  require  a  long  lead 
time,  are  more  likely  to  be  available  when  the  system  is 
fielded . 

Second,  by  developing  estimates  of  training  requirements  for 
the  various  design  alternatives  which  are  considered  during 
early  phases  of  the  acquisition  process,  the  training 
developer  can  provide  the  information  needed  to  effectively 
influence  system  design. 


The  importance  of  obtaining  training  projections  during  the 
earliest  phases  of  system  acquisition  cannot  be 
overestimated.  Most  of  the  major  design  decisions  related  to 
a  new  system  are  made  during  the  early  phases  of  the 
acquisition  process  (see  Figure  1-1).  Thus,  if  training  is 
to  influence  design,  it  must  impact  these  early  design 
decisions.  And  there  is  good  reason  for  ensuring  that 
training-related  considerations  do,  in  fact,  impact 
design . 

In  most  weapon  systems,  manpower  costs,  including  training 
costs,  are  the  largest  component  of  the  system’s  operation 
and  support  costs.  Because  these  costs  are  the  result  of 
demands  generated  by  the  design  characteristics  of  a  system, 
acquisition  policies  have  been  established  by  the  Federal 
Government  to  ensure  that  support  requirements  are 
accurately  determined  and  evaluated  in  conjunction  with 
system  development  (for  example,  the  DoDD  5000  series  on 
major  systems  acquisition).  ETES  is  specifically  designed  to 
provide  the  Army  with  the  capability  for  meeting  the 


training-related  requirements  specified  in  these  acquisition 
policies. 


1.3.1  Requirements  for  an  Early  Training  Estimation  System 

As  part  of  the  first  task  of  the  ETES  development  project, 
an  extensive  review  of  existing  Army  capabilities  for 
conducting  early  training  estimation  was  conducted.  The 
review  identified  three  major  problem  areas  which  limited 
the  Army's  ability  to  develop  early  estimates  of  training 
requirements.  These  problem  areas  are; 


( 1 )  Lack  of  Systematic  Flow  of  Information  Among 
Participants  in  the  Acquisition  Process  -  Under 
current  practices,  training  developers  often  do 
not  receive  information  on  system  functional 
requirements  and  design  concepts  in  any  systematic 
format,  nor  are  they  kept  abreast  of  changes  and 
updates  to  the  design  concepts.  In  addition, 

there  is  also  a  large  amount  of  redundant  data 
development  and  data  collection  among  training 

organizations,  particularly  during  the  early 

phases  of  the  acquisition  process.  Army  system 
developers  and  training  development  organizations 
do  not  currently  have  the  full  capacity  to  share 
information  and  this  can  lead  to  unnecessary 

duplication  of  effort.  As  a  result,  different 

training-related  organizations  often  are  not 
synchronized  with  each  other  or  the  prime 

equipment  developer. 


Lack  of  Estimation  Procedures/Aids  AoDroDriate  to 


the  Early  Design  Process  -  Even  if  training 
developers  receive  accurate  and  timely  information 
on  early  system  concepts,  deficiencies  in  training 
estimation  aids  and  procedures  preclude  the  early 
estimation  of  training  resources.  Most  current 
training  technologies  are  designed  to  be  applied 
with  the  type  of  detailed  data  and  the  types  of 
analytical  questions  which  are  more  relevant  to 
later  phases  of  the  acquisition  process. 


Lack  of  Svstematic  Technolo 


for  Ran  id  1 


Evaluating  Training  Alternatives  -  Currently, 
there  are  no  systematic  procedures  for  rapidly 
assessing  the  training  resources,  cost,  and 
efficiency/effectiveness  of  training  alterna¬ 
tives.  In  addition,  there  are  no  techniques  for 
quickly  conducting  tradeoff  and  sensitivity 
analyses,  and  asking  "what  if"  types  of  questions. 


Trainii 


1.3.2  Intended  User  of  ETES 


As  part  of  the  review  of  existing  Army  procedures  conducted 
during  Task  1  of  the  ETES  development  study,  potential  users 
for  an  early  training  estimation  system  were  identified. 
The  primary  user  organizations  of  ETES  are  expected  to  be 
(1)  the  training  developers  in  the  Army  schools  associated 
with  the  development  of  new  systems,  (2)  program  management 
offices  (PMOs)  for  new  systems,  particularly  those  indi¬ 
viduals  concerned  with  training  development  or  Integrated 
Logistics  Support,  (3)  the  TRADOC  System  Manager  (TSM),  (4) 
other  Army  organizations  concerned  with  training  development 
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such  as  the  TRADOC  System  Analysis  Activity  (TRASANA)  and  PM 
TRADE,  and  (5)  contractors  who  must  develop  training 
requirements  for  new  systems. 


1.4  ETES  STUDY  TASKS 

The  ETES  development  study  involved  five  tasks.  A  summary 
of  these  five  tasks  is  presented  in  Table  1-1.  This  table 
lists  the  subtasks  under  each  task  and  the  major  products 
that  were  produced  under  each  task.  An  overview  of  these 
five  tasks  and  the  original  ETES  concept  is  provided  in  the 
sections  which  follow.  More  detailed  descriptions  of  the 
research  conducted  under  each  of  the  five  study  tasks  is 
presented  in  Sections  2  thru  6. 


1.4.1  Original  ETES  Concept 

In  the  original  ETES  concept  outlined  in  the  statement  of 
work,  ETES  was  to  have  three  components;  a  system 
description  technology,  a  task  generation  procedure,  and  an 
early  system  simulation  model  (see  Table  1-2).  The  system 
description  technology  was  to  consist  of  an  "optimal 
language  for  describing  developing  system  concepts"  and  an 
automated  aid  to  store  emerging  system  concepts  described  by 
the  language.  The  task  generation  procedure  was  to  provide  a 
tool  for  generating  task  descriptions  from  the  early  systan 
descriptions  stored  in  the  system  description  technology. 
The  early  systan  simulation  model  was  to  provide  a  technique 
for  identifying  and  evaluating  critical  tasks.  More 
specifically,  the  simulation  model  would  be  used  to 
determine  how  system  performance  could  be  expected  to  be 
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impacted  by  human  task  perfonnance  and  how  human  task 
performance  would,  in  turn,  be  impacted  by  system  design 
concepts. 

Table  1-2  describes  the  current  ETES  concept  and  conpares  it 
to  the  original  concept.  More  details  in  the  current  ETES 
concept  are  provided  in  Section  1.5.  More  details  on  the 
development  of  the  current  concept  are  provided  in  the 
sections  which  follow. 


1.4.2  Task  1:  Assess  Existing  Concept  Development 
Procedures. 

During  this  task,  three  major  activities  were  accanpl  i  shed . 
First,  Army  Life  Cycle  Management  Model  processes  were 
examined  to  identify  the  key  products  of  an  early  training 
estimation  system  and  to  identify  the  likely  users  of  these 
products.  As  a  result  of  this  effort,  a  detailed  listing  of 
the  documents,  processes  and  output  products  related  to 
early  training  estimation  was  developed. 

Second,  interviews  were  conducted  with  likely  users  of  ETES 
to  more  precisely  define  current  early  training  practices 
and  needs.  These  interviews  indicated  that  early  estimates 
of  training  requirements  (that  is,  estimates  prior  to 
Milestone  1)  were  seldom,  if  ever,  developed.  The 
interviews  were  further  analyzed  to  identify  the  technical 
problems  which  contributed  to  the  lack  of  early  training 
estimation.  Three  major  gaps  were  identified;  (1)  lack  of  a 
tool  for  describing  and  storing  task  and  training 
information,  (2)  lack  of  tools  for  estimating  what  a 
training  program  should  look  like  during  the  early  phases  of 


the  acquisition  process,  and  (3)  lack  of  tools  for 
evaluating  early  training  program  concepts. 

Third,  the  psychological  aspects  of  the  system  design 
process  were  examined  to  identify  the  types  of  problem 
solving  aids  that  might  be  apprqpriate  for  inclusion  in  any 
automated  system  design  aids  that  might  be  developed  under 
ETES. 


1.4.3  Task  2;  Develop  Method  for  System  Concept 
Description 

During  this  task,  five  major  activities  were  accomplished. 
First,  automated  software  requirements  analyses  and  other 
related  tools  were  reviewed.  The  purpose  of  this  review  was 
to  determine  if  any  of  these  tools  would  be  used  as  vehicle 
for  the  system  description  technology.  The  results  of  this 
review  indicated  that  none  of  these  tools  were  appropriate 
because  (a)  they  were  too  complex  to  use,  (b)  they  could  not 
handle  the  frequent  changes  which  occurred  during  the  early 
phases  of  the  acquisition  process,  and  (c)  they  could  not 
describe  all  of  the  types  of  data  elements  and  data  element 
relationships  that  were  required  for  early  training 
estimation. 

When  the  investigation  of  requirements  analysis  tools  proved 
to  be  unfruitful,  a  second  review  was  undertaken  to 
determine  if  an  automated  data  base  management  technique 
could  provide  the  technology  needed  for  early  system 
description.  After  reviewing  off-the-shelf  data  base 
management  systems  for  mainframe  computers,  mini-computers, 
and  microccmputers ,  it  was  determined  that  a  canpr  ehens  ive 


early  system  description  capability  could  best  be  achieved 
by  developing  a  new  microcomupter-based  data  base  management 
system  that  was  specifically  tailored  for  early  system 
design  and  training  program  estimation.  After  examining 
existing  Army  microcomputer  usage  and  the  expected  memory 
requirements  for  an  ETES  data  base  management  systan,  the 
Apple  III  microcanputer  was  selected  as  the  hardware  vehicle 
for  this  tailored  data  base. 

In  the  third  subtask,  recent  research  in  the  area  of  human 
computer  interaction  was  examined  to  identify  general 
guidelines  for  guiding  the  construction  of  SDT  screens, 
human  computer  dialogue  techniques  and  program  logic. 

In  the  fourth  subtask,  the  SDT  was  develqped  in  accordance 
with  the  general  requirements  for  early  training  estimation 
identified  in  Task  1  and  the  human  computer  guidelines 
developed  in  the  previous  subtask. 

In  the  fifth  subtask,  the  SDT  was  applied  to  the  Single 
Channel  Ground  Airborne  Radio  System  (SINCGARS). 

1.4.4  Task  3;  Develop  Training  Estimation  Aids/Procedures 

One  of  the  major  needs  identified  during  Task  1  was  the  need 
for  the  development  of  a  set  of  tools  for  estimating 
training  program  requirements  (tasks  to  be  trained,  course 
requirements,  resource  requirements,  and  costs)  for 
developing  systems.  With  this  in  mind.  Task  3,  which  was 
originally  intended  to  only  develop  procedures  for  task 
generation  was  expanded  to  include  procedures  for  estimating 
(1)  system  functions,  (2)  system  hardwa  re/sof  twa  re  and 


(3)  tasks. 


design  concepts,  (3)  tasks,  (4)  training 
requirements,  (5)  training  resources,  (6)  training 
and  (7)  training  efficiency/effectiveness. 


program 

costs, 


Development  of  the  Training  Estimation  Aids/Procedures  began 
with  construction  of  a  detailed  listing  of  the  functions 
that  were  required  to  estimate  each  of  these  seven  system 
elements.  Existing  tools  which  could  be  modified  to  perform 
these  early  training  program  estimation  functions  were  then 
identified.  Finally,  a  comprehensive  set  of  Training 
Estimation  Aids  and  Procedures  consisting  of  both  modified 
existing  tools  and  newly  developed  tools  was  then 
constructed . 


1.4.5  Task  4:  Develop  Evaluative  Technology 

The  task  was  divided  into  two  subtasks.  During  the  first 
subtask,  existing  simulation  techniques  were  reviewed  to 
determine  if  these  techniques  could  provide  a  systematic 
tool  for  evaluating  emerging  training  concepts.  The  results 
of  the  review  indicated  that  current  simulation  technologies 
were  not  appropriate  for  early  training  estimation  since  (a) 
they  required  data  that  was  unavailable  during  the  early 
phases  of  the  acquisition  process  and  (b)  they  could  not 
provide  output  on  several  of  the  criteria  (e.g.,  resources, 
requirements,  costs)  that  were  needed  to  evaluate  emerging 
training  concepts. 


In  the  second  subtask,  a  listing 
must  be  performed  in  the  early 
program  concepts  was  constructed, 
could  be  modified  to  perform 


of  the  functions  which 
evaluation  of  training 
Existing  tools  which 
these  functions  were 
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identified.  An  integrated  Evaluative  Technology  consistirq 
of  both  modified  existing  tools  and  newly  developed  tools 
was  then  constructed. 


1.4.6  Task  5:  Develop  User's  Guides 

Separate  user's  guides  were  developed  for  each  of  the  ETES 
automated  tools.  In  addition,  the  manual  ETES  procedures 
were  consolidated  into  a  single  guide. 

1.5  CURRENT  ETES  COMPONENTS 

The  current  ETES  has  three  major  components;  a  System 
Description  Technology  (SDT),  Early  Training  Estimation  Aids 
and  Procedures  (TEAP),  and  Evaluative  Technology  (see  Figure 
1-2).  The  SDT  is  a  data  base  management  system  for  storing 
and  tracking  task  and  training-related  data.  The  data  in 
the  SDT  is  used  in  the  TEAP  to  estimate  training 
requirements  for  a  new  system.  These  training  requirements 
include  estimates  of  tasks  requirements,  course 
requirements,  and  resource  requirements  as  well  as  estimates 
of  training  costs,  training  efficiency,  and  training 
"effectiveness".  In  the  Evaluative  Technology,  the 

integrated  impacts  of  training  requirements  are  assessed, 
training  alternatives  are  evaluated,  tradeoffs  and 
sensitivity  analyses  of  key  parameters  are  conducted,  and 
the  relationships  between  ETES  outputs  and  key  Army 
acquisition  documents  and  processes  are  specified. 

More  details  on  the  three  components  of  ETES  are  provided  in 
the  sections  which  follow.  A  detailed  description  of  the 
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SDT  is  provided  in  User's  Guide;  ETES  SDT .  Detailed 
descriptions  of  the  Training  Estimation  Aids  and  Procedures 
and  Evaluative  Technology  are  provided  in  the  ETES 
Procedural  Guide.  Detailed  descriptions  of  three  automated 
tools  associated  with  the  Training  Estimation  Aids  and 
Procedures  and  Evaluative  Technology  are  provided  in  User ' s 
Guide:  Media  Selection  Proaram;  User's  Guide:  Resource  and 


Cost  Est imat ion  Technique;  and  User ' s  Guide: 


Planning  and  Scheduling  Technique. 


Automated 


1.5.1  The  SDT 

The  SDT  is  a  mic  rocanputer-based  data  base  management  system 
for  (1)  describing  actual  and  projected  system  elements 
including  system  functional  requirements,  system  hardware 
and  software  concepts,  tasks,  skills,  and  training  program 
elements  and  their  associated  resource  and  cost 
requirements,  (2)  for  storing  the  above  information,  (3)  for 
changing  and  updating  this  information,  and  (4)  for 
transmitting  the  information  among  the  participants  in  the 
acquisition  process.  The  SDT  data  base  management  system  is 
designed  to  allow  training  developers  to  keep  track  of  the 
numerous  system  changes  which  occur  during  the  early  phases 
of  the  acquisition  process.  In  addition,  it  provides  a 
centralized  data  base,  thus  eliminating  the  redundant  data 
collection  efforts  which  typically  take  place  among  the 
numerous  training  development  organizations  within  the  Army. 

The  computerized  SDT  also  facilitates  the  use  of  automated 
aids  for  estimating  training  program  elements  and  their 
associated  resource  and  cost  requirements.  These  automated 
aids  allow  the  training  develc^er  to  quickly  develop 


training  requirements  estimates  for  alternative  training  and 
system  concepts.  This  capability  greatly  increases  the 
probability  that  training  requirements  will  be  effectively 
considered  during  the  early  phases  of  the  weapon  system 
development  process. 


1.5. 1.1  SDT  Data  Elements 

To  provide  an  effective  communication  vehicle  for  training 
developers  and  other  participants  in  the  acquisition 
process,  the  SDT  has  been  designed  to  describe  (a)  training 
programs  and  their  associated  resources,  (b)  the  tasks  which 
drive  these  training  programs,  (c)  the  personnel  who  will  be 
required  to  perform  the  tasks,  (d)  the  system  designs  which 
generate  the  task  requirements,  and  (e)  the  functional 
requirements  for  which  the  system  designs  have  been 
developed.  An  overview  of  the  major  data  elements  currently 
included  in  the  SDT  is  provided  in  Table  1-3.  Although  the 
data  elements  in  the  SDT  are  designed  to  remain  fixed  for 
any  particular  application,  the  SDT  software  allows  the  SDT 
data  base  manager  to  quickly  change  the  data  elements  to 


meet  the  needs  of  particular  users  and/or  models 
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through  "user-friendly"  human-computer  dialogue  techniques 
such  as  menu  selection  and  question-and  answering. 


There  are  four  major  types  of  variables  in  the  SDT  data 
language : 


Entities  -  Major  system  elements.  Entities  are 
roughly  equivalent  to  nouns  in  the  English 
language.  The  entities  in  the  SDT  are  functional 
requirements,  system  missions,  equipment,  tasks, 
courses,  training  media,  and  personnel. 

Subentities  -  Lower  level  system  elements. 
Subentities  are  linked  to  entities  in  a 


hierarchical  fashion. 


For  example,  "task 


conditions"  are  subentities  of  "tasks." 

Attributes  -  Descriptors  that  delimit  or  specify 
important  properties  of  entities.  Each  attribute 
has  a  set  of  values  associated  with  it. 
Attributes  are  used  to  describe  both  entities  and 
subentities.  For  example,  one  attribute  for  the 
entity  "task"  is  "task  frequency." 


Pointer  Variables  -  Variables  used  to  specify  the 
relationship  which  exist  between  different 
entities,  between  entities  and  subentities  and 
between  entities,  subentities,  and  attributes.  The 
relationships  specified  by  the  pointer  variables 
determine  the  SDT  data  structure.  (The  SDT  uses 
elements  of  both  hierarchical  and  relational  data 


base  structures.) 


1.5.  1.3  SDT  Configuration 


When  fully  implemented,  the  SDT  will  utilize  a  distributed 
processing  architecture  (see  Figure  1-3).  A  centralized 
data  base  for  each  weapon  systan  (or  weapon  system 
alternative)  will  be  stored  on  a  mainframe  computer.  At 
periodic  intervals,  users  will  transfer  a  copy  of  the  data 
base  from  the  mainframe  to  a  local  microcomputer.  Once  on 
the  micro,  users  can  perform  standard  data  base  management 
functions  (input,  output,  modify).  Thus,  all  major  data 
base  management  functions  can  be  performed  independently. 
Once  users  have  completed  their  activities,  they  can 
transfer  the  updated  version  to  the  mainframe.  A  detailed 
"audit  trail"  will  be  kept  for  each  weapon  system  so  that 
users  can  systematically  track  and  assess  system  changes. 
The  current  version  of  the  SDT  is  designed  to  operate  on  ( 1) 
an  APPLE  III  microcomputer  with  a  modem,  printer,  monitor, 
floppy  disk  drive,  and  a  5  megabyte  hard  disk  and  (2)  a 
Honeywell  DPS  8/32  mainframe  computer.  The  mainframe 
computer  is  actually  only  needed  when  there  is  more  than  one 
user  and  these  users  are  located  at  several  different 
sites.  The  SDT  has  built-in  security  features  which  allow 
the  SDT  manager  to  restrict  data  input,  modify,  and  output 
capabilities  to  a  limited  set  of  users. 


1.5.  1.4  SDT  Operation 

To  provide  a  user-friendly  interface,  dialogue  on  the  micro 
computer  relies  on  menu  selection  techniques  for  data  item 
and  command  selection  and  data  output.  Form-filling  and 
question  and  answer  dialogue  techniques  are  used  for  data 
input.  A  "help"  key  is  provided  to  allow  users  to  obtain 
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guidance  on  all  SDT  ccsnmands  and  data  elements.  This  "help" 
capability  can  be  activated  at  any  time  during  the  operation 
of  the  SDT  by  pressing  the  escape  key  on  the  APPLE  keyboard. 


1.5. 1.5  Generation  of  Data  for  SDT 

In  order  to  provide  a  capability  for  early  training 
requirements  estimation,  the  SDT  will  be  used  to  describe 
system  elements  during  the  earliest  phases  of  the 
acquisition  process.  To  generate  data  during  these  early 
phases,  comparability  analysis  procedures,  which  are  part  of 
the  ETES  Training  Estimation  Aids/Procedures,  will  be 
employed . 

During  the  early  phases  of  the  system  acquisition,  when  only 
information  on  a  system's  functional  requirements  is 

available,  comparability  analysis  techniques  will  be  used  to 
identify  existing  subsystems  which  are  similar  to  those  of 
the  new  system.  Historical  data  for  these  subsystems  will 
then  be  collected  and  modified  to  (1)  meet  the  differential 
characteristics  of  the  new  system  and  (2)  correct  any 

inherent  deficiencies  in  the  historical  data  base.  By 
utilizing  design  and  task  data  from  comparable,  existing 
systems,  meaningful  early  estimates  of  training  requirements 
can  be  made  when  only  functional  information  on  the 
projected  system  is  available  (see  Figure  1-4).  Later,  as 
actual  design  concepts  are  developed,  comparability  analysis 
can  be  used  to  develop  estimates  of  tasks  and  training 
program  elements.  Still  later,  when  the  actual  system  tasks 
are  available,  only  the  training  program  elements  must  be 
es t  ima  ted . 


-  Eiliiiiaied  via  Coiii|tarabilily  Analysis 


Thus,  by  adding  comparability  analysis  procedures  to  SDT 
data  base  management  capabilities,  the  SDT  will  be  capable 
of  (1)  describing  alternative  system  concepts  during  the 
earliest  phases  of  the  acquisition  process,  (2)  describing 
projected  system  elements,  (3)  relating  alternative  system 
concepts  to  a  common  framework  so  that  meaningful 
comparisons  can  be  made,  and  (4)  refining  system  information 
as  more  accurate  and  more  detailed  data  are  developed. 

1.5.  1.6  Application  of  SDT  in  the  System  Acquisition 
Process 

In  its  initial  application  to  a  particular  weapon  system 
development  process,  the  SDT  can  be  used  to  describe  the 
system  functional  requirements  which  are  generated  during 
functional  analysis.  These  requirements  specify  the 
functions  which  must  be  performed  if  the  system  is  to 
satisfy  its  designated  mission  need.  The  SDT  can  be 
employed  in  a  functional  analysis  as  soon  as  the  need  for  a 
particular  system  has  been  specified.  Formally,  this 
activity  occurs  after  the  approval  of  the  requirements 
document  at  Milestone  0,  which  initiates  the  Conceptual 
phase  of  the  acquisition  process.  However,  the  SDT  could 
probably  be  used  to  describe  functional  requirements  even 
prior  to  Milestone  0  if  alternative  system  concepts  were 
identified  earlier. 

Once  the  functional  requirements  for  a  system  have  been 
developed  and  described  via  the  SDT,  the  SDT  can  be  used  to 
generate  and  describe  system  designs.  These  designs  specify 
possible  mechanisms  for  performing  the  desired  functions. 
These  mechanisms  include  equipment,  personnel,  and 


Once  developed,  the  system  design(s)  can  also  be  described 
with  the  SDT. 

Once  the  mechanisms  for  accomplishing  the  functions  have 
been  identified  in  the  design  concepts,  the  human  tasks 
which  must  be  performed  to  utilize  the  system  designs  can  be 
specified.  These  tasks,  which  are  the  key  building  blocks  of 
training  development,  can  be  documented  in  the  SDT.  Once 
the  tasks  are  identified  and  specified  in  the  SDT,  training 
estimation  aids  and  procedures  can  be  used  to  determine 
training  program  elements,  estimate  training  resources,  and 
develop  training  products.  The  resulting  training  program 
and  its  associated  resources  can  then  be  documented  in  the 
SDT. 

The  SDT,  like  the  other  components  of  ETES,  is  primarily 
designed  for  applications  during  the  Concept  Exploration 
phase  of  the  acquisition  process,  which  runs  from  Milestone 
0  to  Milestone  1.  However,  the  SDT  can  also  be  used  during 
subsequent  phases  of  the  acquisition  process.  The  primary 
uses  of  the  SDT  applications  during  these  later  phases  are 
to  (1)  make  more  detailed  estimates  of  task  and  training 
resource  requirements,  (2)  determine  the  impact  of 
subsequent  design  changes  on  task  and  training  requirements 
via  the  data  base  management  capabilities  of  the  SDT,  and 
(3)  conduct  trade-off  studies  of  proposed  solutions  to 
identified  training  problems. 

1.5.2  Training  Estimation  Aids/Procedures  (TEAP) 

The  Training  Estimations  Aids/Procedures  are  an  integrated 
set  of  procedures  and  automated  aids  for  performing  six  key 
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early  training  estimation  functions:  (1)  Functional 

Requirements  Analysis  -  Systematic  description  of  the 
functions  which  the  system  must  perfom  and,  where 
necessary,  estimation  of  the  hardware/software  design 
concepts  needed  to  achieve  these  functions,  (2)  Task 
Generation  -  identification  of  the  tasks  required  to  operate 
or  maintain  the  system,  (3)  Training  Program  Estimation  - 
estimation  of  where  and  how  the  tasks  should  be  trained,  (4) 
Training  Resource  Estimation  -  estimation  of  the  training 
resources  needed  to  implement  the  training  program,  (5) 
Training  Cost  Estimation,  and  (6)  Training  Efficiency/ 
"Effectiveness"  Estimation  -  estimation  of  the  adequacy  with 
which  the  training  program  can  be  expected  to  train 
personnel . 

A  listing  of  the  ETES  Training  Estimation  Aids  and 
procedures  is  contained  in  Figure  1-5. 

The  TEAP  includes  new  techniques  as  well  as  existing 
methodologies  from  other  ARI  research  projects  such  as  the 
Training  Efficiency  Estimation  Model  (Jorgensen,  Kubala,  and 
Atlas,  1981),  recent  work  on  cost  and  training  effectiveness 
analysis  by  Dawdy,  Chapman,  and  Frederickson  (1981),  and  the 
Hardware  Procurement  -  Military  Manpower  (HARDMAN) 
methodology  (O'Brien,  1983  ;  Mannle,  1981),  These  existing 
methodologies  were  modified  to  meet  the  specific 
requirements  of  early  training  estimation.  Brief 

descriptions  of  the  Training  Estimation  Aids  and  Procedures 
follow. 
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Figure  1-5.  ETES  Training  Estimation  Aids/Procedures. 


1.5.2.  1  Functional  Requirements  Analysis 

In  order  to  estimate  training  requirements  for  a  new  system, 
the  training  developer  must  have  information  on  the  system's 
hardware/software  design,  and  its  functional  requirements. 
Unfortunately,  this  information  is  often  not  available  to 
training  analysts  during  the  early  phases  of  the  acquisition 
process.  Consequently,  it  is  often  the  case  that  no 
systematic  estimates  of  training  requirements  are  developed 
during  these  phases.  The  TEAP  is  designed  to  assist  the 
training  analyst  in  overcoming  these  problems  by  providing 
systematic  techniques  for  estimating  the  system's  functional 
requirements  and  hardware/software  design  during  the  early 
phases  of  system  acquisition.  The  TEAP  procedures  for 
estimating  system  functions,  and  hardware/software  design 
are  at  a  general  level.  They  are  designed  to  provide  the 
minimum  amount  of  information  needed  for  early  training 
estimation.  As  the  development  of  the  system  progresses, 
and  actual  information  on  system  functional  requirements  and 
hardwa  re/sof  twa  re  design,  are  developed,  these  system 
elements  no  longer  need  to  be  estimated  by  ETES  procedures 
but  can  be  obtained  directly  fron  the  combat  developer 
(functional  requirements)  or  materiel  developer 
(  hardwa  re/sof  twa  re  design  and  manpower  requirements). 

ETES  functional  requirements  analysis  procedures  provide  a 
description  of  the  infomation  which  should  be  generated 
during  the  functional  requirements  analysis  and  the  steps 
that  one  must  go  through  to  generate  these  elements. 

Early  estimates  of  system  hardware/software  design  are 
generated  via  comparability  analysis.  Quite  simply, 

comparability  analysis  involves  (1)  identifying  comparable 


existing  systems,  (2)  collecting  data  on  the  comparable 
systems,  and  (3)  m.odiEying  these  data  to  reflect  the 
differences  between  the  comparable  existing  system(s)  and 
the  new  system. 

The  most  recent  version  of  the  DoD  standard  on  Logistics 
Support  Analysis  (LSA)  ( MI L-STD- 1 388 )  identifies  compar¬ 
ability  analysis  as  the  preferred  method  for  estimating  key 
system  elements  during  the  early  phases  of  the  acquisition 
process . 

Identification  of  comparable  equipment  in  ETES  takes  place 
in  three  major  phases:  (1)  identification  of  the 
Predecessor  equipment  system(s),  (2)  identification  of  the 
Baseline  Comparison  System  (BCS)  equipment  system,  and  (3) 
identification  of  the  New  equipment  system. 2  xhe  BCS  system 
is  constructed  by  (a)  adding  subsystems  to  the  Predecessor 
subsystem  to  reflect  additional  capabilities  required  in  the 
New  system  (i.e.,  the  system  under  development),  and  (b) 
subtracting  subsystems  to  reflect  capabilities  no  longer 
required  in  the  New  system.  The  BCS  provides  a  baseline  for 
comparing  the  New  system  training  requirements  to  other 
similar  systems.  The  BCS  concept  is  directly  congruent 
with  the  new  version  of  MIL-STD-1388  which  specifically 
calls  tor  the  construction  of  a  baseline  comparison  system 
during  the  early  phases  of  the  acquisition  process. 


The  procedures  described  in  this  section  were  derived 
“"rom  the  HARDMAN  Methodology.  A  description  of  the  HARDMAN 
Methodology  is  presented  in  Appendix  D. 


1.5. 2. 2  Task  Generation  Procedures 


In  the  present  version  of  ETES,  comparability  analysis  is 
used  as  the  principal  means  of  estimating  task  requirements. 
In  this  approach,  task  data  for  the  comparable  existing 
system(s)  are  collected  and  modified  to  reflect  the 
differences  in  design  and/or  employment  between  the  New  and 
comparable  systan. 

The  task  generation  process  begins  with  the  collection  of 
task  data  for  the  equipments  and  MOSs  in  the  BCS  system.  The 
BCS  is  canposed  of  the  existing  subsystans  (including  the 
Predecessor  subsystems)  which  come  closest  to  meeting  the 
New  system  functional  requirements.  The  primary  data  source 
for  existing  task  data  is  the  Soldier's  Manual  since  it,  by 
definition,  contains  all  tasks  associated  with  an  MOS.  The 
task  data  from  existing  systems  are  updated  to  reflect  any 
equipment  or  doctrinal  changes  that  were  made  since  the 
source  was  published. 

Once  the  BCS  task  data  has  been  collected  and  updated,  this 
data  is  examined  and  compared  to  the  New  system  hardware/ 
software  design  and  functional  req  ui  rane  nts.  BCS  tasks 
associated  with  equipments  or  functions  which  are  no  longer 
needed  are  eliminated.  New  tasks  are  added  to  reflect  new 
equipment  or  functions.  Descriptions  of  new  tasks  are 
developed  in  accordance  with  existing  Instructional  Systems 
Development  (ISD)  guidelines. 


Identification  of  New  system  operator  tasks  is  facilitated 
by  examining  the  subsystem  functions  identified  during  the 
functional  requirements  analysis. 


By  adding  and  deleting  BCS  tasks,  a  New  system  task  list  can 
be  constructed.  The  reason  for  each  deletion/addition  is 
documented  using  a  systematic  set  of  modification  codes. 

It  is  possible  that  a  New  subsystem  may  require  the  same 

task  as  a  BCS  subsystem  but  that  the  essential  character¬ 
istics  of  this  task  must  be  changed  to  reflect  the  New 

system  requirements.  For  example,  the  same  task  may  be 
required  for  both  the  BCS  and  New  system  but  the  frequency 
with  which  the  task  is  performed  may  differ. 

To  account  for  these  changes  to  essential  task 
characteristics,  all  tasks  with  major  modifications  are 
identified  and  the  reason  for  each  task  modification  is 
documented.  Tasks  associated  with  major  changes  to  skills 
and  knowledges  are  analyzed  further  to  identify  the  specific 
skill  and  knowledge  differences  between  the  BCS  and  New 

system  task.  Tasks  with  minor  changes  in  task  elements  sre 
not  examined  further  (It  is  assumed  that  their  skills  and 

knowledges  have  not  changed).  The  reason  for  this  strategy 
is  that  the  purpose  of  ETES  is  to  estimate  training 
requirements  rather  than  to  develop  training  products  (e.g., 
courses,  manuals).  Hence,  only  major  changes  to  skills  and 
knowledges  which  would  significantly  impact  training 
requirements  are  determined. 


Once  tasks  have  been  identified,  their  conditions  and 
standards  can  be  developed  using  existing  ISD  procedures. 
These  same  procedures  are  used  to  assess  the  adequacy  of  any 
new  task  descriptions  which  are  generated  during 
comparability  analysis. 


1.5. 2. 3  Training  Program  Estimation"^ 


Algorithms  are  provided  to  assist  the  training  analyst  in 
determining  the  tasks  to  be  trained  and  assigning  these 
tasks  to  training  settings.  To  provide  input  to  these 
algorithms,  tasks  are  rated  on  a  series  of  task 
characteristics  (e.g.,  frequency,  learning  difficulty). 
Several  different  algorithms  are  provided  to  meet  the  needs 
of  different  phases  of  the  acquisition  process. 

Ouas i-programs  of  instruction  (QPOI)  are  constructed  by  (1) 
modifying  or  deleting  modules  from  existing  courses  to 
reflect  the  task  deletions  and  modifications  made  during 
task  generation  and  (2)  adding  modules  to  reflect  the  unique 
requirements  of  the  New  system.  As  part  of  the  QPOI 

construction  process,  the  instructional  methods  and 
curriculum  hours  which  must  be  devoted  to  each  module  are 
dete  rmined . 

Media  for  the  training  program  are  selected  by  the 

application  of  an  automated  aid,  the  Media  Selection  and 
Efficiency  Estimation  Program.  This  program  is  an  extension 
of  the  Training  Efficiency  Est-imation  Model  (TEEM)  produced 
by  Jorgensen  et  al .  (1981).  The  Media  Selection  and 

Efficiency  Estimation  Program  significantly  expands  the 
capabilities  of  TEEM  by  recasting  media  selection  as  a 

dynamic  programming  problem  and  automating  these  procedures 
on  the  Apple  III  microcomputer.  This  automated  program 
permits  the  user  to  employ  the  SDT  to  input  and  store  the 


^  The  current  version  of  ETES  only  contains  procedures 
for  estimating  training  programs  for  individual 
institutional  training  courses. 


data  needed  to  feed  the  media  selection  procedures.  Using 
dynamic  programming  techniques,  the  program  can  assign  tasks 
to  media  in  a  manner  that  optimizes  efficiency,  relative 
cost,  or  efficiency  and  relative  cost. 

Efficiency  is  detemined  by  canparing  the  stimulus, 
response,  and  feedback  characteristics  of  the  individual 
task  to  the  stimulus,  response,  and  feedback  characteristics 
of  potential  media  categories.  More  specifically,  a  score 
is  calculated  which  describes  the  match  between  media  and 
task  characteristics.  Efficiency  for  each  task-media 
combination  is  calculated  by  dividing  this  score  by  the 
maximum  match  that  may  be  achieved  for  the  task. 

Total  efficiency  for  a  set  of  tasks  is  detemined  by 
aggregating  the  efficiency  score  for  individual  tasks.  An 
additional  efficiency  measure  can  be  calculated  by  weighting 
the  efficiency  of  each  task  by  the  task  criticality  score. 
This  task  criticality  score  is  calculated  by  aggregating  the 
task  factors  typically  used  in  selecting  tasks  for  training 
(e.g.,  task  frequency,  percent  members  perfoming,  task 
delay  tolerance,  etc.). 


1.5. 2. 4  Estimation  of  Training  Resources 

These  procedures  estimate  the  training  resources  needed  to 
implement  the  training  program.  The  training  resources 
encompassed  by  these  procedures  include  (a)  the  number  of 
students  to  be  trained,  (b)  number  of  instructors  and 
support  personnel,  (c)  facilities  requirements,  (d)  training 
device  and  training  equipment  requirements,  and  (e) 
ammunition  requirements.  Included  among  the  procedures  are 


techniques  for  using  off-the-shelf  automated  worksheet 
software  (e.g.,  VisiCalc)  for  storing  and  applying  several 
of  the  resource  determination  algorithms. 


1.5. 2.5  Estimation  of  Training  Costs 

These  procedures  estimate  the  costs  of  the  resource 
requirements  and  aggregate  these  costs  into  a  total  course 
cost.  The  procedures  include  techniques  for  using  off-the- 
shelf  automated  worksheet  software  to  calculate  course 
costs.  The  procedures  also  describe  how  to  use  modified  data 
from  comparable  existing  courses  to  assist  in  the  cost 
estimation  process. 


1.5. 2. 6  Training  Ef  f  ic  i  ency/Ef  feet  ive  ness  Estimation 

Procedures  are  provided  for  determining  the  training 
efficiency  of  selected  aspects  of  the  training  program 
(e.g.,  media).  Training  efficiency  is  defined  as  a  measure 
of  the  extent  to  which  the  characteristics  of  the  training 
program  element  match  the  task  characteristics  of  the  New 
system.  For  example,  media  training  efficiency  is 

determined  by  comparing  the  stimulus,  response,  and  feedback 
characteristics  of  the  tasks  to  the  stimulus,  response,  and 
feedback  characteristics  of  potential  media  categories.  An 
additional  variant  of  this  efficiency  measure  may  be 
obtained  by  weighting  each  task  by  its  "criticality"  where 
criticality  is  determined  by  aggregating  weighted  scores  of 
key  task  characteristics  (e.g,,  frequency,  consequences  of 
inadequate  pe  rf  o  ima  nee )  . 


Training  "effectiveness"  is  determined  by  using  modified 
versions  of  procedures  outlined  in  Dawdy,  et  al  (1981).  More 
specifically,  a  group  of  subject  matter  experts  is  presented 
with  the  following  information  for  each  task:  (a)  the  target 
population  description  of  the  personnel  who  will  perform  the 
task,  (b)  a  description  of  the  task  including  its  associated 
conditions,  job  performance  standards,  and  general  skills 
and  knowledges,  (c)  a  description  of  the  training  program  or 
training  program  elements  (e.g.,  course  length,  methods, 
media)  that  will  be  used  to  train  the  task,  and  (d)  the 
criterion  which  must  be  achieved  for  the  task  (if  different 
from  the  job  performance  measure).  Each  subject  matter 
expert  is  then  asked  to  estimate  the  expected  percentage  of 
soldiers  in  the  target  population  who  will  pass  the 
criterion  given  that  training  program. 4 

It  should  be  noted  that  the  ETES  TEAP  is  designed  to 
estimate  training  requirements  for  a  New  system.  The  TEAP 
is  not  designed  to  provide  techniques  for  actually 
developing  instructional  materials  or  programs  (e.g., 
course,  instructor  packages,  student  packages,  training 
literature,  training  devices).  Thus,  the  TEAP  will  assist 
training  analysts  in  determining  what  instructional 
materials/programs  should  be  produced,  what  the  content  of 
these  materials/  programs  should  be,  and  in  estimating  the 
cost  of  these  materials/programs. 


I  It  is  recognized  that  this  measure  of  effectiveness, 

may  differ  from  other  conceptualizations  of  training 
effectiveness.  It  is  also  recognized  that  this  approach 
only  provides  a  gross  measure  of  "effectiveness"  that  should 
only  be  used  during  the  earliest  phases  of  the  acquisition 
process . 

I 
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This  focus  on  training  requirements  estimation  rather  than 
on  training  product  development  is  one  of  the  unique 
characteristics  of  the  TEAP.  This  focus  on  training 
requirements  distinguishes  it  from  other  existing  training 
development  methodologies  such  as  the  Array's  Instructional 
Systems  Development  (ISD)  process  and  the  Army  Research 
Institute's  Training  Develcper's  Decision  Aid  (Hawley, 
1979) . 

The  training  requirements  produced  by  the  TEAP  provide 
front-end  information  needed  for  the  early  planning  and 
analysis  of  training  programs.  The  training  requirements 
information  produced  by  the  TEAP  provides  the  foundation  for 
the  actual  construction  of  training  products  in  ISD  and 
other  related  methodologies.  Techniques  and  methodologies 
which  can  be  used  to  develop  training  products  are  provided 
in  TRADOC  Pam  350-30,  Schulz  and  Farrell  (1980),  Hawley 
(  1979),  and  Fink  (1981).  An  overview  of  the  relationship 
between  ISD  and  ETES  is  provided  in  Figure  1-6. 


1.5.3  Evaluative  Technology 

The  Evaluative  Technology  is  an  integrated  set  of  procedures 
and  automated  tools  for  (1)  developing  f  igures-of-mer i t  for 
assessing  the  integrated  impacts  of  the  training  require¬ 
ments  developed  in  the  Training  Estimation  Aids/Procedures, 
(2)  identifying  potential  problem  areas  for  system  training 
and  the  likely  sources  of  these  problems,  (3)  identifying/ 
evaluating  training  problems,  (4)  developing  training- 
related  input  to  key  acquisition  documents,  and  (5) 
determining/evaluating  training  development  schedules  (see 


^JT  ^'" 


ISO  BLOCKS  IN  EACH  OF  THE  FIVE  ISO  PHASES 

THE  BLOCKS  IN  EACH  PHASE  ARE: 


PHASE! 


r^' 


PHASE  it 


-►: 


PHASE  III  —  ► 


PHASE  IV  —  ► 


PHASE  V 


I.l 

analyze 

JOS 

X 

1.2 

W  SELECT 

^  tasks/ 

FUNCTIONS 

X 

1.3 

V  CONSTRUCT 

^  JOE 

^  RERRORMANCE 

measures 

X 

1  i 

W  analyze 

P  existing 
COURSES 

X 

IS 

W  select 

F  instructional 
SFITINC 

X 

yir 

n.i 

OEVELOR 

OKJECTIVCS 

^2.2 
^  OEVELOR 
^  TESTS 

a. 3 

W  DESCRIBE 
^  ENTRY 

behavior 

X 

a. 4 

W  determine 

F  SEQUENCE  k 

structure 

X 

— 

▼ 

B.i 

SREOEY 

learning 

EVENTS/ 

activities 

A 

m3 

SREOEY 

^  INSTRUCTION 
m  management 
^  RLAN* 
DELIVERY 
SYSTEM  A 

m.3 

^REVIEW /select 
r  existing 
materials 

.  m  4 
m DEVELOR 
'  instruction 

k.  ®  5 
p  validate 

T  instruction 

B  1 

IM^EMCNT 

»  instructional 

manacemcnt 

R\,AN 


B.2 

.  CONOUCT 

instruction 


2  ' 

1  2.2  1 

W  CONOUCT 

i  ^  CONOUCT  !  1 

F  internal 

i  F  external  1  1 

Evaluation 

1  evaluation  1 

2  3 

>  REVISE 
SYSTEM 


X  =  Function  Covered  by  ETES 


FIGURE  1-6.  RELATIONSHIP  BETWEEN  ISO  AND  ETES 


Figure  1-7).  A  summary  of  the  procedures  in  each  of  these 
areas  is  provided  below. 


1.5.3.  1  Development  of  Figures-of -Merit 

Procedures  are  included  for  identifying  f igures-of-mer i t 
which  summarize  the  essential  features  of  the  training 
requirements.  Eight  potential  training  f igures-of-mer it  are 
utilized  including  (1)  cost,  (2)  training  efficiency,  (3) 
training  "effectiveness",  (4)  congruence  with  training 
development  guidelines,  (5)  congruence  with  program 
requirements,  (6)  training  complexity,  (7)  training 
capacity,  and  (8)  feasibility.  In  addition,  a  procedure  is 
provided  for  constructing  a  summary  evaluation  score  which 
aggregates  the  scores  on  the  individual  f  igur  es-of-meri  t. 
This  summary  evaluation  score  provides  a  global  measure  of 
the  "goodness"  of  a  training  program. 


1.5. 3. 2  Identification  of  Problem  Areas 

Procedures  are  included  for  identifying  training  problem 
areas.  These  problem  areas  consist  of  the  Army  Military 
Occupational  Specialties  (MOSs)  which  have  high  figure-of- 
merit  scores  relative  to  (1)  the  Predecessor  system,  and/or 
(2)  the  other  MOSs  in  the  New  system.  Procedures  are  also 
provided  for  identifying  the  courses,  equipments,  and  tasks 
which  contribute  to  the  high  f  igur  es-of -me  r  i  t  and 
identifying  the  training  program  elements  which  are  likely 
causes  of  the  high  f  igure-of-me  ri  t  scores. 


1.5. 3. 3  Identif ication  and  Evaluation  of  Alternatives 


Guidelines  are  included  for  (1)  identifying  training 
alternatives  which  can  address  the  training  problems,  (2) 
evaluating  the  training  alternatives  through  selected 
reapplication  of  TEAP  and  Evaluative  Technology  procedures, 

(3)  conducting  sensitivity  analyses  of  key  paraneters,  and 

(4)  assessing  the  impact  of  non-training  system  changes 
(e.g.,  hardware/software  changes)  on  training. 


1.5. 3. 4  Development  and  Evaluation  of  Training 
Schedules /Plans 

Construction  of  training  development  schedules  for  emerging 
systems  is  a  difficult  process.  Over  100  developmental 
events  are  listed  in  TRADOC  Reg  351-9,  the  Army  regulation 
governing  training  plan  development.  The  sequential 
relationships  among  these  events  are  complex  and  are  not 
described  in  any  systematic  and  integrated  manner  in  TRADOC 
Reg  351-9.  To  assist  users  in  developing  training 

schedules,  procedures  are  provided  for  using  automated,  off- 
the-shelf  scheduling  software  (e.g.,  Vi  s  iSchedule )  to  track 
and  monitor  the  training  development  schedule.  By  using 
this  software,  the  training  developer  can  quickly  and 
efficiently  respond  to  changes  in  the  training  development 
schedule.  Use  of  off-the-shelf  scheduling  software  is 
facilitated  by  the  inclusion  of  an  input  data  diskette  v^ich 
(a)  describes  the  events  ir.  the  training  development  process 
(as  specified  in  TRADOC  Reg  351-9),  (2)  describes  the 
temporal/sequential  relationships  among  these  events  and  key 
acquisition  milestones,  and  (3)  lists  the  expected  duration 
of  these  events  for  a  "typical"  major  Army  weapons  system. 
This  data  diskette  significantly  reduces  data  input 
requirements.  In  addition,  it  eliminates  the  need  for  an 
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analysis  of  the  canplex  sequential  relationships  among 
training  development  events  which  are  either  implicitly  or 
explicitly  specified  in  TRADOC  351-9. 

1.5. 3.5  Develop  Inputs  to  Acquisition  Processes  and 
Documents 

Major  Army  acquisition  processes  and  documents  related  to 
early  training  estimation  were  identified  and  are  summarized 
in  Table  1-4  . 

1.6  RELATIONSHIP  BETWEEN  ETES  AND  OTHER  ARI  PROJECTS 

ETES  is  part  of  the  Army  Research  Institute's  thrust  to 
develop  systematic  techniques  for  determining  manpower, 
personnel,  and  training  requirements  for  developing  Army 
systems.  Other  major  projects  currently  in  this  thrust 
include  the  Man-Integrated  Systems  Technology  (MIST),  the 
Army  Manpower  and  Personnel  Requirements  Estimation  Program 
(ARMPREP)  ,  the  Army  Hardware  Procurement-Military  Manpower 
(HARDMAN)  program,  and  the  Human  Resources  Test  and 
Evaluation  System  (HRTES).  MIST  is  the  central  project  in 
this  thrust  since  it  will  integrate  the  other  projects  in 
the  thrust.  Table  1-5  summarizes  the  data  that  each 

component  will  contribute  to  the  three  phases  of  MIST 
development.  ETES  will  provide  MIST  with  two  critical 
components:  (1)  teohniques  for  estimating  and  evaluating 

training  requirements  and  (2)  a  prototype  data  base 
management  systan  (the  SDT)  for  storing  MPT  information. 

A  description  of  the  other  projects  in  the  ARI  thrust  is 
provided  in  Appendix  D. 


Table  1-4  Major  Army  Acquisition  Processes  and  Documents 
Related  to  Early  Training  Estimation 

Cost  and  Training  Effectiveness  Analysis  (CTEA) 

Outline  Individual  and  Collective  Training  Plan  (OICTP/ICTP) 
Logistics  Support  Analysis  (LSA) 

Operational  Testing  (OT) 

Training  Device  Requirements,  Documents,  and  Processes 

-  Training  Device  Letter  of  Agreement  (TDLOA) 

-  Training  Device  Requirements  (TDR) 

-  Training  Device  Letter  Requirement  (TDRL) 

-  Training  Device  Study  (TDS) 

New  Equipment  Training 
System  Requirements /Documents 

-  Justification  for  Major  System  New  Start  (JMSNS) 

Letter  of  Agreement  (LOA) 

Request  for  Proposal  (RFP)  Development /Evaluation 
Personnel  Documents/Processes 

-  Tentative  Quantitative  and  Qualitative  Personnel 

Requirements  Information  (TQQPRI) 

Integrated  Personnel  Summary  (IPS) 

System  Level  Documents/Processes 

Concept  Formulation  Package  (CFP) 

Cost  and  Operational  Effectiveness  Analyses  (COEA) 

-  Tradeoff  Determination  (TOD) 

Best  Technical  Approach  (BTA) 

Decision  Coordinating  Paper  (DCP) 

Defense  System  Acquisition  Review  Council  (DSARC) 

Army  System  Acquisition  Review  Council  (ASARC) 

Mission  Area  Analysis 


SECTION  2 


TASK  1:  ASSESS  EXISTING  CONCEPT 
DEVELOPMENT  PROCEDURES 


During  this  task,  three  major  activities  were  accomplished. 
First,  Army  Life  Cycle  System  Management  Model  processes  and 
documents  were  examined  to  identify  the  products  that  must 
be  developed  by  an  early  training  estimation  system  and  to 
identify  the  likely  users  of  these  products.  Second, 
interviews  were  conducted  with  likely  users  of  the  SDT  to 
more  precisely  define  current  early  training  estimation 
practices  and  needs.  Third,  the  psychological  aspects  of 
the  design  process  were  examined  to  identify  problem  solving 
aids  which  might  be  included  in  any  automated  aids  developed 
in  ETES. 

2.1  ASSESS  LCSMM  INTERFACES 

During  this  tasK,  a  detailed  review  was  conducted  of 
existing  Army  and  DoD  documents  relating  to  system 
acquisition  or  training  development.  The  purpose  of  this 
review  was  to  (1)  identify  current  requirements  for  early 
training  estimation,  (2)  identify  likely  data  elements  for 
inclusion  in  the  SDT,  and  (3)  identify  any  tools  currently 
being  used  for  early  training  estimation.  The  review 
focused  on  documents  and  processes  related  to  Mission  Area 
Analyses  (Pre-Milestone  0),  Concept  Exploration,  and 
Demonstration  and  Validation  phases  of  the  acquisition 
process.  The  ETES  review  of  training-related  acquisition 
processes  was  the  major  input  for  a  later  report  describing 
manpower,  personnel  and  training  interfaces  in  the  Army 


LCSMM  produced  by  Wagner  (1982)  as  part  of  the  Man- 
Integrated  Systems  Technology  (MIST)  project. 

Table  2-1  describes  the  documents  and  processes  which  were 
identified  as  having  the  most  relevance  to  early  training 
estimation. 

The  two  documents/processes  which  were  determined  to  be  the 
most  critical  to  early  training  estimation  were  (1)  the 

Outline  Individual  and  Collective  Training  Plan  (OICTP)  and 
(2)  Cost  and  Training  Effectiveness  Analysis  (CTEA). 

2,1.1  OICTP/ICTP 

The  OICTP  and  ICTP  are  plans  which  support  the  development 
and  implementation  of  new  or  revised  individual  and 

collective  training  programs  at  institutional  and  unit 
levels.  The  OICTP  and  ICTP  are  the  major  resource  and 
planning  documents  for  developing  training  for  new  Army 

systems.  An  approved  OICTP  is  sufficient  justification  to 
enter  manpower  and  funding  reguirements  into  the  Army's 
programming  and  budgeting  processes  for  inclusion  of  the 
TRADOC  Review  of  Manpower  (TRM).  As  specified  in  existing 
regulations  (TRADOC  Reg  351-9),  the  OICTP/ICTP  contains  a 
proposed  training  concept  and  training  strategy.  The 

concept  and  strategy  are  typically  generated  by  the 
proponent  school  responsible  for  the  New  system's  training 
r equ i rements . 

The  OICTP/ICTP  is  generally  an  evolving  document  that 
increases  in  specificity  (via  appropriate  up>lates)  as  the 
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system  under  development  is  further  defined.  The  OICTP/ICTP 
provides  significant  feeder  data  for  the  Tentative 
Qualitative  and  Quantitative  Personnel  Requirements 
Information  (TQQPRI),  the  Training  Effectiveness  Analysis 
(TEA),  and  the  New  Equipment  Training  Plan  (NETP).  After 
LOA  approval,  the  OICTP  can  be  used  by  the  contractor  to 
facilitate  the  identification  of  training  requirements 
conducted  as  part  of  the  logistic  support  analysis. 

The  CICTP/ICTP  is  intended  to  guide  the  development  of 
training  subsystem  requirements  and  to  provide  the  general 
framework  for  their  future  incorporation  into  the  existing 
training  base.  The  regulations  specifically  emphasize  the 
intention  to  use  the  proposed  training  concept  in  the 

OICTP/ICTP  to  "...  identify  the  constraints  which  training 

requirements  and  resources  may  impose  on  the  design  of  the 
material  system"  (TRADOC  Reg  351-9,  pg .  5).  The  regulation 
also  mandates  use  of  the  OICTP/ICTP  as  the  vehicle  to 

describe  "...the  integration  of  the  training  subsystem  into 
the  development  of  the  total  system  and  the  integration  of 
the  developing  system  into  ongoing  training  systems." 

TRADOC  Reg  3  51-9  further  stipulates  that  the  OICTP/ICTP  must 
incorporate  the  principles  of  Army  Training  1990  (AT  90) 
into  training  for  a  new  system,  for  both  institutional  and 
unit  training  and  at  all  skill  levels  for  the  MOS/SC 

affected.  The  OICTP/ICTP  is  intended  to  develop  and 
describe  a  systematic  and  feasible  strategy  for  training, 
extending  from  the  development  of  "initial  qualification"  to 
continuing  "sustainment  of  the  proficiencies"  needed  for  the 
successful  fielding  and  operational  deployment  of  the  system 
being  acquired. 


As  specified  in  existing  regulations,  the  OICTP/ICTP 
provides  information  on  the  training  required  to  integrate 
replacements  fran  the  training  base  into  the  unit,  and  to 
qualify  personnel  for  higher  level  tasks  as  they  advance  in 
grade.  The  OICTP/ICTP  is  further  directed  to  provide 
information  on  the  identification,  quantification,  and  need 
for  training  devices,  simulators,  documentation/ 
publications,  training  aids,  support  facilities, 
instructors,  costs,  and  all  other  support  and  logistic 
considerations  necessary  for  the  implementation  and  test  of 
the  proposed  training  plan. 

Table  2-2  provides  an  overview  of  the  elements  in  and 
OICTP/ICTP  and  describes  which  ETES  procedures  can  provide 
input  to  the  development  of  these  elements. 


2.1.2  The  Cost  and  Training  Effectiveness  Analysis  (CTEA) 

The  CTEA  process  is  potentially  the  most  critical  LCSMM 
process  related  to  early  training  estimation  since  it 
requires  the  user  to  not  only  estimate  what  the  training 
program  will  look  like  (as  does  the  OICTP/ICTP),  but  also  to 
evaluate  these  training  programs  and  to  provide  training 
related  input  into  the  overall  system  development  and 
evaluation  process. 

The  new  TRADOC  Reg  3  51-9,  governing  the  OICTP  process, 
contains  the  most  current  definition  of  the  CTEA.  It 
defines  CTEA  as  "a  methodology  which  involves  documented 
investigation  of  the  comparative  effectiveness  and  costs  of 
alternative  training  systems  for  attaining  defined 
performance  objectives."  The  definition  further  specifies 


Covered  In  tTES 


RELATEO  r'<R  FRUCEDURCS/MDS 


that  a  CTEA  can  focus  on  any  one  or  a  combination  of  the 
following: 


Training  concepts 
Training  strategies 
Training  equipment/devices 
Programs  of  instruction 
Training  impacts  of 

-  New  materiel 

-  Organization 

-  Tactics 

-  Families  of  systems 


Regardless  of  the  specific  focus  of  the  CTEA,  TRADOC 
Regulation  351-9  stipulates  that  the  CTEA  should  include  an 
analysis  of  the  attainable  levels  of  proficiency  and  the 
costs  associated  with  each  alternative.  In  addition,  a  CTEA 
should  include  a  cost  effectiveness  trade-off  analysis  of 
the  feasible  alternatives.  This  regulation  further 
specifies  that  a  CTEA  must: 


o  Ensure  that  training  development  is  initiated 
early  in  the  life  cycle  of  hardware  systems  and  is 
accomplished  in  coordination  with  combat 
developments , 

o  Optimize  soldier-hardware  subsystem  interface, 

o  Ensure  that  all  feasible  training  subsystem 
alternatives  are  considered, 

o  Optimize  soldier-training  subsystem  interface. 


o  Optimize  soldier-training  subsystem  interface, 

o  Recommend  the  preferred  training  alternative  based 
on  cost  and  training  effectiveness,  and 

o  Provide  decision  makers  with  more  precise 
information  at  critical  points  in  the  acquisition 
process  concerning  the  total  system  (comprising 
the  training,  hardware,  and  other  subsystems). 

These  objectives  demonstrate  that  the  CTEA,  unlike  the  OICTP 
and  the  QQPRI/BOIP,  is  an  early  training-related  document 
intended,  in  theory  at  least,  to  influence  the  hardware 
system  design.  This  is  a  very  significant  difference  from 
the  OICTP  which  is,  by  regulation,  primarily  an  MPT  planning 
and  resource  document.  The  steps  in  the  CTEA  process  and 
the  corresponding  ETES  procedures  are  detailed  in  Table  2-3. 

2.1.3  Identification  of  LCSMM  Reports 

As  part  of  the  review  of  existing  LCSMM  processes,  an 
attempt  was  made  to  identify  any  report  formats  which  might 
be  relevant  to  early  training  estimation,  and  in  particular, 
to  the  identification  of  data  elements  for  the  SDT.  Table 
2-4  summarizes  the  report  formats  that  were  identified 
during  this  process.  A  listing  of  the  documents  that  were 
reviewed  during  this  process  is  contained  in  Appendix  E. 
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Table  2-4  Summary  of  Standardized  Reports  Related 
To  Early  Training  Estimation 


Title  Document  I  terns 

Task  List,  TRADOC  CIR  351-8  Task  Number,  Task 

Training  Setting  Description, 

Assignments  Training  Materials, 

Responsibility  for 
Training 

TRADOC  CIR  350-3  Task  No., 

Task  Description, 

No.  of  Duty  Positions, 
Training  Pro(jucts,  Supporting 
Tasks 

TCH  Computation  TRADOC  CIR  351-12  Course  Module,  Instructional 
Sheet  Hours,  Method,  Instructor 

Contact  Hours 

Equipment 
Nomenclature 
Part  Number,  Task  No., 

Task  Description 


DI-H-0001 


Task  Analysis  DI-H~0006 

For  Manning  (MIL-STD-XYZ ) 

Report  DI-H-0004 


MOS  Training 

Program 

Worksheet 


Task  Inventory 
Repor  t 


Mission,  Scenario, 
Conditions,  Function, 
Job/Duty  Position 


Table  2-4  (continued) 


Title  Document  I  terns 

Task  List  MIL-M-63040 (TM)  JPG/ETM,  Equipment,  MOS, 

Item  Code,  Tasks/Subtasks, 
Conditions,  Standards, 
Comments 


Task  Selection 

TRADOC  PAM 

Task  No;  Task  Description 

Sheet 

351-4  (T) 

Task  Characteristics/ 

F'actor  s 

Job  and  Task 

TRADOC  PAM 

Task  No. ,  Task 

Analysis 

351-4  (T) 

Description;  Condition, 

(Worksheet) 

TRADOC  REG  351-4 

Standard,  Job  Title, 

Supervisory  Job  Reqs., 

Task  Characteristics, 
Enabling  Skills  and 
Knowledge,  Equipment  Used 
to  Per  form  Task . 

Task  Analysis  ARI  Research  MOS  Skill  Level,  Task 

Worksheet  Product  80-13*  Number,  Task,  Initiating 

Cues,  Conditions,  Standards 
Reference/Tips 

Task  Summary  ARI  Research  Task,  Conditions, 

Sheet  Product  80-13  Standards,  References, 

Performance  Description 

Task  Information  TRADOC  CIR  381-12  Task  Number,  Title, 

Sheet  Standard 


Course 
Summa  ry 


TRADOC  CIR  381-12 


t.'ourse,  Hours 


Table  2-4  (continued) 


Title 

Document 

I  terns 

Task  Cluster 

Annex 

TRADOC  CIR  351-12 

Task  Cluster,  Hours, 

Student/Instructor  Ratio 

Lesson  Objective,  Lesson 

Reference 

Training  Resource 

Sect  ion 

TRADOC  CIR  351-12 

Course  Number,  Title, 

Manpower  Requirements, 

Expense  Elements 

MOS  Course 

TRADOC  Cost 

Course  Costs  by 

Costs 

Analysis  Program 

Cour  ses  in  MOS 

Cour  se 

Costs 

ATRM-159 

Course  Title,  Course 

Number,  MOS,  Course  Cost 

Elements 

Equ ipment 

List 

MIL-M-63035 (TM) 

ESN,  Part  Number,  FSCM, 

Descript  ion 

Tool  and 

Test  Equipment 

MIL-M-63035 (TM) 

Maintenance  Level, 

Nomenclature,  National 

Stock  Number 

Ma  1  n  r  t--' na  nee 

A  i  1  oca  1 1  <in 

MIL-M-63035 (TM) 

Group  Number,  Component, 

Maintenance  Function, 

Maintenance  Level,  Tools 
and  Test  Equipment  No. 


'  Cha  r  t ) 


Table  2-4  (continued) 


Title  Document 

Preliminary  MIL-M-63035 (TM) 

Task  Development 

Worksheet 


Training  TRADOC  REG  351-9 

Subsystem 

Summary 

Sheet 


LSAR  Input  MIL  STD  1388-1 

Data  Sheets 
A  through  J 


BOIP  p-'eeder  AR  71-2 

Data  Shieet 

QQPRI  Report  DARCOM  P-750-16 

Pa  t  t  1 
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Items 

Task,  Equipment,  Tools 
and  Test  Equipment, 

Personnel  Reqs.,  Equipment 
Condition,  References, 
Preliminary  Tasks,  Follow-on 
Task,  Task  Elements 

System  Data,  Development 
Schedules  for  Following 
Training  Products  -  Technical 
Manuals,  SM/TG/JB,  ARTEP , 
Resident  Training  Program, 
Resident  Training  Equipment, 
Devices,  Training  Literature, 
TEC,  Audiovisual,  ACCP, 
Facilities/Ranges,  Ammunition, 
NET.  Also,  Resource 
Requirements  by  Fiscal  Year. 

End  Item  Maintenance 
Requirements,  Reliability 
and  Maintainability,  Summary 
of  Maintenance  Task  Data, 
Maintenance  and  Operator 
Tasks,  Training  Materiel 
Description,  Facility 
Description,  Skill  Evaluation, 
Supply  Support  Requirements 

Description  of  End  Items  to 
be  Entered  into  Inventory, 
Items  to  be  Replaced 

Description  of  MOS ,  Qualified 
or  to  be  Qualified  During  the 
Development  Process 


2.2  INTERVIEW  ACQUISITION  PARTICIPANTS 

The  review  of  the  LCSMM  documents  identified  prescribed 
requirements  pretaining  to  the  area  of  early  training 
estimation.  To  identify  what  was  actually  done,  interviews 
were  conducted  with  several  of  the  likely  participants  in 
early  training  estimation.  Table  2-5  summarizes  the 
organizations  that  were  contacted  during  these  interviews. 
The  major  finding  of  these  interviews  was  that  little,  if 
anything,  is  typically  done  in  the  area  of  early  training 
estimation  prior  to  Milestone  1.  The  only  activity  which 
takes  place  during  this  period  is  the  inclusion  of  some  pro 
forma  statements  of  training  requirements  which  are  included 
in  system  requirements  documents  such  as  the  Letter  of 
Agreement  (LOA)  despite  the  fact  that  several  regulations, 
particularly  the  newer  regulations,  such  as  TRADOC  Reg  3  51- 
9,  indicated  that  several  training-related  activities  should 
take  place  during  this  time  frame.  Three  reasons  were 
identified  for  this  lack  of  early  training  estimation:^ 


Lack  of 


Systematic  Flow  of  Information  Amonc 


Participants  in  the  Acquisition  Process  -  To  develop 
estimates  of  training  requirements,  training  developers 
require  information  on  actual  or  estimated  systans 
functional  requirements  and  design  concepts  as  soon  as 
they  are  generated,  and  to  maintain  the  accuracy  of 
these  estimates,  these  same  training  developers  must  be 
quickly  informed  of  design  changes  and  updates.  A 


A  more  recent  review  by  Adolman  et  al  . 
indicated  that  these  problems  still  exist. 


19  82)  has 
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Table  2-5.  Organizations  Interviewed  During  Task  1 


Location 

Air  Defense  School, 
Fort  Bliss,  Texas 


Armor  School, 

Fort  Knox,  Kentucky 


Redstone  Arsenal, 
Huntsville,  Alabama 


Army  Training  Support 
Center  (ATSC) 

Fort  Eustis,  Virginia 

Training  Developments 
I nst itute , 

Fort  Monroe,  Virginia 

Signal  School , 

Fort  Gordon,  Georgia 


TRASANA , 

White  Sands  Missile  Range 


•  Director  of  Training  Developments 

•  Director  of  Combat  Developments 

•  Division  Chief,  Short  Range  Air 

Defense 

•  Division  &  Assistant  Division  Chief, 

Patriot  and  Nike  Hercules 

•  Patriot  Project  Manager  for  Training 

•  Educational  Specialist,  ROLAND 

•  Educational  Specialist,  SPAs 

•  ROLAND  Project  Officer,  DTD 

•  Associate  TSM,  Patriot 

•  Associate  TSM,  Roland 

•  Associate  TSM,  DIVAP 


•  Associate  TSM,  XMI 

•  Staff  of  Directorate  of  Combat 

Developments 

•  Staff  of  Directorate  of  Training 

Developments 


•  Chief  of  New  Equipment  Training, 

Patriot 

•  Chief  of  Training  Developments, 

Patriot 

•  Chief,  New  Equipment  Training  Branch 

•  Chief  of  Training  Developments, 

ROLAND 


•  Staff  of  ATSC 


•  Training  Specialist  Overseeing 

Transfer  of  Information  on  New 
Training  Technologies 

•  Director  of  New  Equipment  Analysis 

Division  and  Staff 

•  Director  of  Combat  Developments 

and  Staff 

•  Training  Analysts,  Directorate  of 

Training  Development 

•  Staff,  Training  Effectiveness 

Analysis  Division 


Table  2-5  (continued) 


Location 


Organizations/ Individuals  Interviewed 


Army  Logistics  Management 
Center , 

Port  Lee,  Virginia 
DARCOM,  HQ 

Alexandria,  Virginia 

MRSA,  Lexongton-Bl uegrass 
Army  Depot,  Kentucky 


Staff,  LAMC 


Staff  of  Associated  Directorage 
for  ILS 

Staff  of  Logistics  Engineering 
Branch 

Director  of  Human  Dimensions 
and  Staff 

Staff  of  Deputy  Chief  of  Staff 
for  Training  (DCST) 


TRADOC,  HQ 

Fort  Monroe,  Virginia 


functional  requirement  defines  what  a  particular  design 
must  do  to  accomplish  a  specific  mission  need. 
Unfortunately,  under  current  practices  and  procedures, 
training  developers  do  not  receive  information  on 
system  functional  requirements  and  design  concepts  in 
any  systematic  format,  nor  is  there  any  formal 
mechanism  through  which  they  can  obtain  information  on 
system  updates. 


In  addition  to  the  information  flow  problems  which  cut 
across  functional  areas,  there  were  additional  information 
flow  problems  among  individual  training  organizations.  More 
specifically,  there  was  a  large  amount  of  redundant  data 
development  and  data  collection  among  training  organiza¬ 
tions,  particularly  during  the  early  phases  of  the 
acquisition  process.  System  developers  and  TRADOC 

organizations  did  not  have  the  capacity  to  share 
information.  This  led  to  unnecessary  duplication  of 
effort.  As  a  result,  different  training-related 

organizations  often  were  not  "singing  from  the  same  sheet  of 


music . 


Lack  of  Estimation  Procedures/Aids  Appropriate  to  the 
Design  Process  -  Even  if  training  developers  were 
receiving  accurate  and  timely  information  on  early 
system  concepts,  current  deficiencies  in  training 
estimation  aids  and  procedures  preclude  the  early 
estimation  of  training  resources.  Current  training 
technologies  are  designed  to  be  applied  with  the  type 
of  detailed  data  and  the  types  of  analytical  questions 
which  are  relevant  to  later  phases  of  the  acquisition 


process.  These  technologies  cannot  address  the  special 
requirements  of  the  early  phases  such  as  the 
idercif ication  of  comparable  existing  equipment,  the 
generation  of  tasks  for  systems  whose  hardware  has  not 
yet  been  built,  and  the  rapid  estimation  of  training 
reso.’.rces  and  costs  and  other  evaluation  criteria. 


Lack  of  Systematic  Technol( 


for  Rapidly  Evaluatim 


Training  Alternatives  -  Currently,  there  are  no 
systematic  procedures  for  rapidly  assessing  the 
training  resources,  cost,  and  ©f f iciency/ef fectiveness 
of  training  alternatives.  In  addition,  there  are  no 
techniques  for  quickly  conducting  trade-off  and 
sensitivity  analyses,  and  asking  “what  if“  types  of 
questions . 


2.3  REVIEW  PSYCHOLOGICAL  ASPECTS  OF  DESIGN  PROCESS 

During  this  subtask,  psychological  research  relating  to 
engineering  design  was  reviewed.  The  goal  of  the  review  was 
to  identify  general  guidelines  for  SDT  software  development. 
The  revie.  focused  on  the  individual  cognitive  processes 
involved  in  system  design  and  attempted  to  identify  features 
that  should  be  included  in  the  SDT  to  facilitate  the  system 
jevelopment  process. 

The  review  began  with  the  search  for  a  general  cognitive 
mode-,  which  could  be  used  to  describe  the  design  process. 
Ramsey  and  Atwood's  (1979)  scheme  for  describing  the  problem 
solving  process  was  determined  to  be  the  most  suitable 
framework.  Using  the  framework  provided  by  this  scheme, 
prob.'.em  solving  aids  relevant  to  each  stage  of  the  problem 


solving  process  were  identified  through  a  review  of  the 
problem  solving  literature.  The  implications  that  each  of 
these  limitations  had  for  the  SDT  were  then  assessed  and 
documented.  A  summary  of  this  analysis  is  provided  in  Table 
2-6.  While  it  was  possible  to  incorporate  some  of  the 
features  of  the  problem  solving  aids  into  ETES  automated 
tools,  in  many  cases  it  was  found  that  the  problem  solving 
aids  were  not  relevant  to  the  types  of  procedures  which  were 
eventually  included  in  ETES. 


Aidin9 

Mftchanism 


Ahirnativt 

Evaluation 


Altarnativa 

Gantratton 


Automatic  Action 
Execution 


Automatic 

Takeover 


Setter  Weiqntmq 
of  Unreliable 
Data 

Cfianqe  of 

Problem 

Reoretentalion 


Daemon 

Coniiiteocv 


Oeciiion  Sfrateqv 


Table  2-6  PHOBLEM-SOLVING  AIDS  RELATED  TO  ETES* 


Theee  aide  may  either  automate 
the  uaar'i  evaluation  critaria,  require 
the  user  to  use  ettabiished  criteria, 
or  simulate  the  results  of  actions 
that  cio  r>ot  have  wall  established 
evaluation  criteria. 

These  aids  are  primarily  used  to 
qenerate  altarnativas  that  the  user 
would  not  normally  consider  or, 
for  extremely  well-defirtad  tasks, 
to  present  alqorithmically 
determined  aittmatiees. 


Such  aids  permit  the  user  to  neme 
the  desired  action  without 
explicitly  carrying  out  the  steps 
involved  m  its  execution. 


This  type  of  aid  furtctions  as  an 
automated  decision  maker  thet  is 
able  to  select  alternative  actions  on 
the  basis  of  prior  observations  of  the 
human  decision  maker's  behavior 
Although  allocatton  of  control  to 
this  aid  occurs  automatically,  when¬ 
ever  some  criterion  of  cofrespOft> 
dance  between  predicted  and  observed 
human  behavior  is  reached,  voluntary 
turnover  of  control  n  aleo  possible. 
Such  an  aid  allows  the  problem 
solver  to  'undo"  the  effects  of 
recent  actions  and  return  to  an 
earlier  state  of  tha  problerrvsolving 
prooasa  without  actually  starting 
over 

This  aid  re-cDdes  low-fidelity  date 
into  a  form  thet  is  more  readily 
usable  by  the  problem  solver. 

Typical  imolementations  of  this 
aid  present  problems  as  <somoronic 
variations  of  mora  itatsdard  oroblem 
reoresantationi.  It  is  mtanded  that 
this  will  aid  the  problem  solver  m 
selecting  an  aoprooriete  and 
pffiaent  oroblem  formulation 
This  type  of  aid  assists  (ha  users 
applyirtg  their  own  deosion 
strategies  consistently  -n  cases  m 
vwhich  these  strategies  are 
comoiex 

Such  aids  assist  the  user  >n 
applyirsg  proOlem-soiving 
techniques  that  would  not 
normally  be  considered  or 
known 


Oecompositioo  ino 
Recomposition 


Oiiruot'on  of 
l*svrhologicai  Set 


This  type  of  jid  allows  the  user  to 
jividt  (he  original  problem  mto  sub¬ 
problems  The  solutions  of  the 
various  suboroblemi  are  tneo 
r:ombinetl  into  a  solution  to  the 
original  'arger  problem 
Such  an  iid  is  intended  to  disrupt 
iny  Mias  or  ’sets  '  that  me  user 
may  employ  irsd  thereby  stimulate 
more  creetive  or  novel  probfem- 


Ramsey  ind  Atwood 


Except  for  aidi  that  automata  tha 
user's  evaluation  criteria,  thtse  task 
aids  ara  task-tpeafic.  Mon  usaful  if 
the  task  is  not  well-eiefined  or  if  a 
large  number  of  evaluation  critarie 
need  be  oonssderad. 

Except  for  weil^iefined  task  domains, 
where  they  may  have  very  little 
impact,  they  c<e  difficult  to 
connruet.  Can  be  con-eff active  for 
training  applications,  but  generally 
are  of  iinutad  use  m  complex 
profaiam-solving  tasks. 

Mott  useful  when  the  results  of 
applying  an  acoon  do  rsot  impact 
subsequent  problam-solving  actions. 

If  this  is  the  case,  tha  user  may 
need  sophisticated  alternative 
evaluation  heuristics. 

Although  demonstrated  to  he 
affective  in  some  contexts  It.g., 
control  tasksl.  the  range  of  tasks 
in  which  this  IS  appropriate  >s  not 
wall  understood.  User  acceptance 
may  ba  low  and  should  be 
carefully  axamined. 


Useful  in  tasks  where  it  is 
pouibie  to  "undo"  recent  actions. 
Can  improve  performance  at 
refepveiv  little  development  con. 


Depends  on  the  ability  to 
accurately  recode  low-fidelity  data. 


Mon  uatful  in  wall-understood 
tasks  An  inappropriate 
representation  rnay  sariouslv 
degrade  performence. 


Useful  for  sxoert  problem  solvers 
•n  ivell-defined  tasks  including 
sufficient  vertettlity  ro  adapt  to 
individual  users  rnay  be  difficult. 

Useful  >n  well-defined  tasks  -n 
which  optimal,  or  near  ootimcl 
arobiem-solving  lechniques  are  known, 
oi  *n  tasks  m  which  general 
heurifftics.  such  as  problem 
vedvction.  are  appl'coble.  Reouirts 
detailed  knowledge  oi  the  task 
Useful  only  <1  a  task  can  be 
decomposed  mto  •ndvoendvnt 
subprobiems  Requires  a  good 
understanding  of  rhe  task 


Rotentiallv  uselol  but  may 
disrupt  an  appropriate  set 
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SECTION  3  -  TASK  2:  DEVELOP  METHOD  FOR 
SYSTEM  CONCEPT  DESCRIPTION 


During  this  task,  five  major  activities  were  accomplished. 
First,  as  recommended  in  the  statement  of  work,  automated 
soft'  are  requirements  analysis  tools  were  reviewed  to 
determine  if  any  of  these  tools  could  be  used  to  develop  and 
document  early  system  descriptions.  When  this  review  proved 
unfruitful,  automated  data  base  management  tools  were  then 
examined.  This  analysis  did  prove  to  be  fruitful  as 
evidenced  in  the  current  SDT  which  is  a  microcomputer-based 
data  base  management  system. 

Third,  recent  research  in  the  area  of  human  computer 
interaction  was  examined  to  identify  general  guidelines 
which  could  be  used  in  the  development  of  the  SDT. 

Fourth,  the  SDT  was  developed  in  accordance  with  the  general 
requirements  for  early  training  estimation  identified  in 
Task  1  and  the  human-computer  guidelines  developed  in  the 
third  subtask. 

Fifth,  the  SDT  was  applied  to  the  Single  Channel  Ground 
Activated  Radar  System  (SINCGARS).  More  details  on  each  of 
these  five  activities  are  presented  in  the  sections  which 
follow. 


3.1  REVIEW  REQUIREMENTS  ANALYSIS  TOOLS 


The  review  of  requirements  analysis  tools  was  conducted  in  a 
three-stage  process  by  DRC’s  software  engineering  group. 
During  the  first  stage,  DRC  surveyed  government  reports, 
IEEE  Software  Engineering  Transactions,  and  other  trade 
publications  to  determine  what  tools  were  available  in  the 
area  of  requirements  analysis.  Fortunately,  a  ccmpr  ehens  ive 
review  of  requirements  analysis  tools  had  just  been 
completed  by  Devorkin  and  Obenodorf  (19  79).  Further 

investigation  indicated  that  this  report  contained  all 
requirements  analysis  tools  with  sufficient  maturity  for 
possible  use  in  the  SDT. 

During  the  second  phase  of  the  review,  the  methodologies 
listed  in  Devorkin  and  Obendorf  were  reviewed  in  more 
detail.  Each  review  began  with  an  examination  of  the 
available  literature  on  the  methodology.  Following  the 
literature  review,  individual  users  were  interviewed  by 
phone.  With  the  aid  of  user  comments  and  knowledge  of  the 
SDT  requirements,  criteria  were  developed  for  identifying 
methodologies  with  a  high  degree  of  potential  application  to 
the  SDT.  The  evaluation  criteria  were  as  follows: 


1.  Appl icab  i  li ty .  The  methodology  must  be  capable  of 
building  a  data  base  of  the  conceptual  information 
normally  available  during  the  early  phases  of  a 
developing  or  evolving  system.  This  data  base  must  be 
capable  of  refinement  as  more  specific  system 
information  becomes  available.  It  must  be  capable  of 
describing  requirements,  design  concepts,  human  tasks, 
and  training  pro'jram  elements. 


2. 


Unde  rs  tandabi  1  i  ty  .  The  methodology  must  be  capable  of 
being  understood  by  the  types  of  "personnel"  who  are 
likely  to  use  the  SDT. 

3.  Demons  tr atabi  1  i  ty  .  The  methodology  must  have  been 
applied  to  a  number  of  different  types  of  projects. 

4.  Tr  a  ns  po  r  t  ab  i  1  i  ty  .  The  methodology  must  be  capable  of 
being  implemented  at  a  minimum  of  cost  on  standard 
business  processors  used  in  military/  government 
agencies . 

5.  Training  .  The  methodology  must  have  an  existing  formal 
training  program  available  to  potential  users. 

6.  Sponsors  nip.  The  methodology  must  have  a  specific 
government  agency,  university  or  industry  committed  to 
enhancing  the  methodology  to  meet  additional  user  needs 
as  they  become  known.  The  methodology  must  reside  in 
the  public  domain. 


While  investigating  the  first  few  methodologies,  it  became 
evident  that  there  were  two  main  thrusts  in  the  area  of 
requirements  definition  methodologies.  One  thrust 

emphasized  graphics  representation,  primarily  through 
functional  flow  block  diagrams,  as  a  means  of  specifying 
relationships  between  system  elements.  Another  thrust 
emphasized  a  high  level  conceptual  language  as  the  mechanism 
for  specifying  relationships  between  these  system  elements. 
Because  there  was  good  deal  of  overlap  between  the  tools 
within  each  of  these  two  thrusts,  particularly  among  the 
language-based  tools  which  are  ail  basically  more  advanced 


derivatives  of  earlier  work  conducted  by  the  ISDOS  project 
at  the  University  of  Michigan,  it  was  decided  that  the  tools 
listed  in  Devorken  and  Obendorf  would  be  evaluated  in  terms 
of  the  six  criteria  listed  above,  and  that  the  tool  in  each 
of  the  two  major  thrust  areas  with  the  highest  evaluations 
on  these  criteria  would  be  selected  for  further  analysis  in 
the  third  stage  of  the  review. 

Table  3-1  displays  the  requirements  analysis  tools  which 
were  evaluated  during  this  stage  and  summarizes  their 
assessme  nt . 

The  two  tools  selected  for  further  analysis  were  the  ICAM 
Definition  Language  or  IDEF,  which  was  determined  to  be  the 
best  graphics-based  tool,  and  the  Problem  Statment 
Language/Problem  Statement  Analyzer,  which  was  selected  as 
the  best  language-based  tool. 

During  the  third  stage  of  the  review,  these  two  tools  were 
examined  in  even  greater  detail.  As  this  review  was  being 
conducted,  it  became  apparent  that  the  requiranents  analysis 
could  not  provide  the  vehicle  needed  for  the  SDT.  There 
were  two  reasons  for  this. 

First,  the  requirements  tools  did  not  appear  to  be  suited 
for  the  types  of  uninitiated  users  who  would  utilize  the 
SOT,  This  was  not  surprising  when  one  considers  that  all  of 
these  tools  were  specifically  designed  to  describe  software 
raj  ui  reme  nts  for  large  complex  systems.  Hence,  they  are 
designee  to  he  utilized  by  technical  personnel  who  have 
fairly  sophisticated  backgrounds  in  software  development. 
'The  tools  were  designed  by  software  sjiecialists  tor 
software  s  [le  c  i  ai  i  s  ts  )  . 


At  a  slightly  more  conceptual  level,  another  factor 
contributing  to  the  complexity  of  the  requirements  analysis 
tools  is  that  they  are  designed  to  be  extremely  flexible 
tools  //hich  can  be  utilized  to  describe  a  ny  type  of 
system.  This  type  of  flexibility  necessitates  a  certain 
degree  of  abstractness.  This  high  degree  of  flexibility  and 
its  associated  abstractness  may  actually  be  a  hindrance  in 
describing  data  for  developing  weapons  systems  since  such 
data  tends  to  be  similar  across  systems. 

Second,  while  requirements  analysis  tools  deal  with  an 
important  aspect  of  early  training  estimation  (i.e.,  system 
description),  they  are  not  really  geared  for  dealing  with 
other  important  ETES  related  problems;  namely,  the  update 
and  refinement  of  these  system  descriptions  and  their 
communication  to  a  wide  range  of  participants  in  the 
acquisition  process 

3.2  REVIEW  DATA  BASE  TOOLS 

When  the  investigation  of  requirements  analysis  tools  proved 
tj  be  unfruitful,  the  focus  turned  to  automated  data  base 
management  systems  (DBMSs),  as  a  mechanism  for  storing  and 
'iescribing  developing  system  descriptions.  Generally,  DBMSs 
fulfill  the  SDT  evaluation  criteria  of  applicability, 
unde rs ta nda b i 1 i ty  ,  demons tra ta b i  1  i ty  ,  t ra nspor ta b 1 1 i ty  , 
training,  and  sponsorship  that  were  identified  in  Section 
3.1.  In  addition,  most  DBMS  have  the  following  advantages 


1.  The  DBMSs  are  designed  to  store  many  data  items 

that  are  related  to  one  another  The  SDT  consists 
of  many  data  items  with  complex  interrelation¬ 

ships.  Therefore,  DBMS  technology  facilitates  the 
development  of  the  SDT. 

2.  DBMSs  allow  data  to  be  centrally  located  and 

controlled.  This  simplifies  data  sharing  among 
multiple  users. 

3.  They  can  be  fitted  with  data  access  aids  that  are 

easy  to  use.  These  aids  allow  a  user  to  input, 

modify,  delete,  and  output  data  using  "user- 
friendly"  techniques  such  as  English-like  phrases 
and  commands. 

4.  Access  to  data  items  can  be  restricted, 

unauthorized  users  cannot  view,  modify,  delete,  or 
output  restricted  data  items. 

5.  The  format  of  a  data  item  is  independent  of  the 
computer  program  that  is  accessing  it.  This  is 
significant  if  future  software  systems--other  than 
the  SDT--wish  to  access  the  data  in  the  SDT  data 
base . 

3,2.1  Overview  of  DBMS  Concepts 

Before  examining  the  results  of  the  review  of  DBMSs,  it  may 
be  helpful  to  review  definitions  and  k-"/  f>aatjres  of  the 
DBMS  concept. 


An  automated  data  base  management  system  may  be  defined  as  a 
computerized  and  integrated  collection  of  stored  operational 
data  used  by  the  applications  groups  of  a  particular 
enterprise!.  The  key  word  in  this  definition  is 

"integrated."  The  data  elements  of  a  system  are  likely  to 

have  relationships  or  associations  which  one  could  use  to 

link  these  elements  to  one  another.  A  data  base  is 
integrated  when  it  incorporates  information  on  these 
relationships  as  well  as  information  on  the  data  elements 
themselves.  This  information  can  be  used  to  store  and 

retrieve  data.  It  should  be  noted  that,  strictly  speaking, 
a  data  base  need  not  be  resident  in  a  computer  or  its 

associated  media.  However,  all  automated  data  bases  will  be 
stored  on  a  computer  or  related  media  and  all  modern  DBMSs 
are  automated.  It  is  clear  that  only  an  automated  data  base 
can  meet  the  storage  and  retrieval  requirements  of  the 
SDT.  The  term  "operational  data"  is  used  to  refer  to  data 
which  is  pertinent  to  the  ongoing  activities  of  an 
enterprise.  Operational  data  excludes  input  data,  work 
queues,  output  data  (such  as  messages  or  reports)  or  any 
other  form  of  temporary  information. 

o  Advantages  of  a  Data  Base  Management  System 

The  major  advantage  of  a  data  base  management  system  (DBMS) 
is  that  it  provides  the  enterprise  with  integrated, 
centralized  control  of  its  operational  data.  This 

centralized  control  can,  in  turn,  (1)  reduce  redundancy  in 
stored  data,  (2)  avoid  inconsistency  in  stored  data,  (3) 


^  This  definition  is  a  mod i f i oa t i on  of  an 
definition  by  Engels  (1971). 
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allow  for  greater  sharing  of  data,  (4)  permit  standards  to 
be  enforced,  (5)  permit  security  restrictions  to  be  applied, 
(6)  permit  a  greater  capability  for  checking  and  maintaining 
data,  and  (7)  provide  "data  independence."  Data 
independence  is  achieved  by  maintaining  an  internal 
structure  of  the  data  which  is  independent  of  the  individual 
applications  of  the  data  and  individual  user  viewpoints. 
This  data  independence  may  be  contrasted  with  data  dependent 
systems  in  which  the  way  data  is  stored  and  the  way  it  is 
accessed  are  dictated  by  the  structure  of  the  applications. 

o  Data  Base  Management  System  Features 

Perhaps  the  best  way  to  describe  the  essential  features  of  a 
DBMS  is  to  outline  an  "architecture"  for  a  typical  DBMS. 
Such  an  architecture  is  displayed  in  Figure  3-1.  This 
architecture  is  taken  directly  from  Date  (1977).  DBMS 
architectures  are  typically  divided  into  three  general 
levels:  internal,  conceptual,  and  external.  The  internal 
level  is  concerned  with  the  way  in  which  the  data  is 
actually  stored  physically.  The  external  level  reflects  the 
users'  view  of  the  data.  The  conceptual  level  provides  the 
medium  for  linking  the  internal  and  external  views.  The 
conceptual  model  provides  a  genera^  community  view  of  the 
data  base  since  it  contains  an  abstract  representation  of 
the  entire  data  base.  This  community  view  is  to  be 
contrasted  with  the  external  views  of  individual  users  who 
typically  will  only  have  a  view  of  a  portion  of  the  data 
base . 


Perhaps  another  way  to  describe  these  three  different  levels 
of  a  DBMS  is  to  refer  to  the  structures  that  psycho¬ 
linguistics  uses  to  describe  human  language.  The  external 
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view  of  a  DBMS  can  be  construed  as  being  roughly  analogous 
to  what  psycholinguistics  describe  as  the  "surface 
structure"  of  language,  while  the  conceptual  level  can  be 
construed  as  being  analogous  to  the  "deep  structure"  of 
language  and  the  internal  structure  can  be  construed  as 
being  roughly  analogous  to  the  physical  structures  in  the 
brain  for  representing  speech. 

It  is  possible  for  each  external  user  to  have  his  own 
"language"  for  utilizing  the  data  base,  although  in  many 
cases,  all  or  a  large  number  of  users  can  use  the  same 
la  nguage . 


The  conceptual  model  is  defined  by  a  conceptual  schema  which 
includes  a  definition  of  each  of  the  various  types  of 
conceptual  information  in  terms  of  content  only  (storage  or 
access  features  are  not  described).  Thus,  the  conceptual 
model  provides  the  definition  of  the  total  data  base 
content.  The  conceptual  model  is  critical  in  that  all  other 
aspects  of  the  DBMS  are  affected  by  the  conceptual  model. 
It  has  a  major  effect  on  the  format  and  structure  of  the 
data  sublanguage  which  is  used  to  store,  update,  and 
retrieve  information  from  the  data  base. 


3.2.2  Process  and  Results  of  Reviewing  Data  Base  Management 
Techniques 

Review  of  data  base  management  systems  for  the  SDT  occurred 
in  a  four  step  process.  First,  existing  mainframe/ 
microcomputer  data  base  management  systems  were  reviewed. 
This  review  did  not  identify  a  data  base  which  could  meet 
ETFS  requirements.  Second,  existing  mic ro-compu t e r  based 


data  base  management  systems  were  reviewed.  While  no  off- 
the-shelf  microcomputer  based  data  base  management  system 
was  identified  which  could  meet  the  ETES  requirements,  a 
concept  was  developed  for  developing  a  tailored  micro¬ 
computer-based  data  base  management  system  which  could 
specifically  meet  ETES  requirements.  Third,  hardware/ 
software  was  identified  for  implementing  this  concept. 
Fourth,  concurrent  with  the  review  of  mainframe  databases 
existing  military  data  bases  containing  training  related 
information  was  reviewed  to  assess  their  applicability  to 
the  SDT.  None  of  these  existing  data  bases  met  ETES 
requi rements . 


3.2.2.  I  Review  of  Mainframe/Microcomputer  Based  DBMSs 

Fifty-one  (51)  commercially-ava liable  mainframe/mini¬ 
computer  based  DBMSs  were  surveyed  through  a  study  of  DATA 
PRO  reports  {70E-0lB-61a  and  D30-100-002)  and  other 
literature.  Table  3-2  summarizes  the  characteristics  of  the 
51  DBMSs  that  were  surveyed.  Each  data  base  is  described  in 
terms  of  six  basic  characteristics:  Vendor  of  the  DBMS, 

Supporting  Hardware,  Approximate  Usage,  Primary  Data 
Organization,  Approximate  Price,  and  Applicability  to  SDT. 

"Vendor  of  the  DBMS"  refers  to  the  company  that  distributes 
the  DBMS.  "Supporting  Hardware"  refers  to  the  computer 
hardware  configuration  on  which  the  DBMS  will  operate. 

"Approximate  Usage"  is  an  estimate  of  the  number  of 
installations  of  the  DBMS.  It  is  based  on  the  following 

s  c  -i  1  e  : 


Table  3-2  Characteristics  of  Commerciai iy  Available 
Main-Frame/Mini-Computer  Based  DBMSs, 


DBMS 

Vendor  of 
the  DBMS 

Supportinq 

Hardware 

Approximate 

Usage 

Primary 

Data 

Organization 

Approx. 

Price 

Applicability  to 
SDT 

'AOABAS 

Software  AG  of  North 

America 

IBM;  360,370. 

303x.  4300 

Moderate 

Network 

S2S00/ 

month 

S40-160K 

Very  high 

AOMINS/II 

Admins,  Inc. 

DEC:  POP-11. 
VAX-11 

Very  low 

Relational 

N/A 

Low 

AMBASE 

Amoour  Computer 
Company 

DEC;  POP-11 

N/A** 

N/A 

S16.5K 

Moderate 

BASIS 

Battelle,  Columbus 
Laboratories 

IBM;  COC;  Cyber; 
Univac;  DEC 

Very  low 

Relational-like 

S38K 

Moderate 

CREATE 

Complete  Computer 
Systems 

Data  General: 

..OVA  &  Eclipse 

Very  low 

N/A 

$18K 

Moderate 

'DATACOM/OB 

Applied  Data  Research « 
Inc. 

IBM:360,370 

Low 

Relational 

S47-57K 

Very  high 

DATA  DEMON 

Gemini  Information 
Systems.  Inc, 

Perkin-Elinsr; 

IBM:  Saries/1 

Very  -jw 

N/A 

$1000/ 

month 

$17.5K 

Low 

OBM-I 

Condor  Computer  Corp. 

CronMnoo: 

Syslem/3 

Very  low 

N/A 

S10K 

Low 

DBMS 

Prime  Computer,  Inc. 

Prinw;  400  &  500 

N/A 

Network 

S20K 

Low 

DBMS  2 

EGS  Systems,  Inc. 

Modular  Computer; 
MOOCOMP 

Very  low 

N/A 

N/A 

Very  low 

□BMS-10 

DEC 

DEC;  System-10 

Low 

Network 

S30K 

High 

■ 

DBMS-11 

DEC 

DEC:  PDP-11 

N/A 

Network 

S16.5K 

Low 

DBMS- 20 

DEC 

DEC:  System- 20 

Very  low 

Network 

S30K 

High 

OBMS-300 

Compudata  Systems,  Inc. 

DEC;  300  Senes 

Very  low 

N/A 

SI  00/ mo 
S5K 

Very  low 

DBMS-990 

Texas  Instruments.  Inc. 

Tl:  DS990. 

Models  6  3i  8 

N/A 

N/A 

S2K 

Very  low 

DL/I  DOS/VS 

IBM 

IBM:  370, 

303*.  4300 

High 

Hierarchical 

S434/mo 

High 

DMS  II 

Burroughs  Corp. 

Burroughs:  3 

700  or  800 

Low 

Network 

S23.25K 

High 

DMS  90 

Sperry  Univac 

Univac:  Series 

30  or  90 

Very  low 

Network 

N/A 

Moderate 

DMS-170 

Control  Data  Corp. 

COC:  6000.'  Cyber 

70,  170,  700 

Very  low 

Network 

S730/mo 

High 

DMS-1100 

Sperry  Univac 

Univac:  1100 

Senes 

Moderate 

Network 

N/A 

High 

DMS/1700 

Dedicated  Systems.  Inc 

Burroughs:  81700 

N/A 

N/A 

S5K 

Low 

DNA-Data 

Basa  Manaqer 

Exact  Systems  & 
Programming  Corp. 

DG;  Nova  or 

Eclipse 

Very  low 

N/A 

N/A 

Very  low 

•DBMS  $ai«cted  for  further  study. 
••Not  available. 
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Table  3-2  (continued) 


OBMS 

Vendor  of 
the  DBMS 

Supporting 

Hardware 

Approximate 

Usage 

Primary 

Data 

Organization 

Approx. 

Price 

Applicability 

SDT 

DPL 

National  Information 
Systems,  Inc. 

DEC:  System 

10  or  20 

Low 

Hierarchical 

S966/mo 

S38-47K 

High 

ORS/XBS 

A.R.A.P. 

IBM.  DEC.  Univac, 
CDC 

Low 

Network 

S22-60K 

High 

EASE  DBMS 

Bloodstock  Computer 
Services,  Inc. 

DEC:  POP-11 

Very  tow 

N/A 

$6.5K 

Low 

GIS/2 

IBM 

IBM;  360  or 

370 

N/A 

Hi^nrchical 

SS20-970/ 

mo 

Moderate 

IBOB 

Teseract  Corp. 

IBM:  Saries/1 

Very  low 

N/A 

$4.1  K 

Low 

•IDS-I/II 

(OM-IV) 

Honeywell  Info. 

Systems,  Inc. 

Honeywell:  Series 
6000  &  60  Level  66 

High 

Network 

S400-S2K/ 

mo 

Very  high 

*IOMS 

Cullinarw  Corp. 

IBM;  360.370, 

303x,  &  4300 

Moderate 

Network 

$50K/yr 

Very  high 

IDOL 

Science  Management 
Corporation 

Wang:  2200; 

IBM:  Series  1 

Low 

N/A 

$364/ mo 
$9500 

Moderate 

IMS 

IBM 

IBM:  360,370. 

303x.  4300 

Vary  high 

Hierarchical 

$1045/ 

mo 

High 

Infoflax  OBM 

Imeraetive  Info. 

Systems,  Inc. 

DEC;  Datasystem 

500  Series 

Very  tow 

N/A 

$12K 

Low 

INFOMEOIA 

Mead  Technology 
Laboratories 

IBM;  360  or  370 

N/A 

N/A 

$2000/ 

mo 

$140K 

Low 

INFOTRIEVE 

Educational  Data 

Systems 

EOS:  Point  4; 

OG :  Nova 

Very  tow 

n/a 

$2000 

Moderate 

INGRES 

INGRES,  Inc. 

DEC;  PDP-11 

Low 

Relational 

N/A 

Low 

IQ/NET 

Infodata  Systems,  Inc. 

IBM:  4300 

N/A 

Network 

S40K 

Moderate 

INQUIRE 

Infodata  Systems,  Inc. 

IBM-  360,  370 

Low 

Network 

$70-1 50K 

High 

MADMAN 

G.E.  Company 

DEC  PDP-11 

N/A 

Relational-tike 

$20  K 

Low 

MIOMS 

National  Technical 

Info.  Service 

IBM:  360 

N/A 

N/A 

$450 

Moderate 

MINOS 

Minnesota  Oatasystems, 

Inc. 

BTI:  4000, 

5000.  or  3000 

Very  low 

N/A 

$3-5K 

Low 

•MODEL  204 

Computer  Corp.  of 

America 

IBM:  360.  370. 

303x.  or  4300 

Very  tow 

Network 

$90-1 50  K 

Very  high 

OASIS 

University  of  Windsor 

IBM:  360.  370. 
or  303x 

Very  low 

N/A 

$30K/yr 

Low 

ORACLE 

Relational  Software.  Inc. 

DEC:  POP11  or 

VAX 

N/A 

Relational 

S43-96K 

Jerate 

OS  200  OB 

Honeywell 

Honeywell'  200 
or  2000 

Very  low 

N/A 

Bundled 

free  with 

Hardware 

Very  low 

PLUSM 

Century  Analysis,  Inc. 

NCR:  101  or 

above 

N/A 

N/A 

S10K 

Low 

QCRT 

The  Management  Group, 

fnc. 

IBM-  360  or  370; 
Honeywell;  6000 

Very  low 

N/A 

S25K 

Moderate 

o  Very  high  -  greater  than  1,500  installations, 


o  High  -  1000  to  1,499  installations. 


o  Moderate  -  500  to  999  installations. 


o  Low  -  100  to  499  installations,  and 


o  Very  low  -  less  than  100  installations, 


It  infers  a  rough  measure  of  the  popularity  of  a  DBMS  within 
the  computer-used  community.  However,  this  inference  does 
not  imply  that  a  greatly  used  system  is  better  than  one  of 
less  usage. 

"Primary  Data  Organization"  is  the  logical  structure  of  the 
data  base  at  the  conceptual  level.  The  structure  can  be 
relational,  network,  hierarchical,  or  a  combination  of  these 
three.  However,  only  the  most  commonly  referenced  structure 
IS  listed. 

"Price"  refers  to  the  basic  system  purchase  price  that  was 
quoted  to  DATA  PRO  in  late  1980.  The  purchase  price 
generally  includes  an  unspecified  monthly  maintenance 
charge.  Monthly  or  yearly  lease/rental  plans  are 
occasionally  included. 


column  entitled  "Applicability  to  SDT"  is  a  composite  of 
the  12  SDT  Requirements  that  were  identified  for  a  mainframe 
data  base  management  system.  Each  DBMS  was  examined  to 
determine  how  many  of  these  12  requirements  it  fulfilled. 
The  scale  used  was  as  follows: 


o  Very  high  -  All  12  requirements  fulfilled, 


o  High  -  9  to  1 1  requirements  fulfilled, 


o  Moderate  -  5  to  8  requirements  fulfilled. 


o  Low  -  3  to  4  requirements  fulfilled,  a  nd 


o  Very  low  -  2  or  fewer  requirements  fulfilled. 


The  applicability  factor  is  the  deciding  factor  for 
selecting  DBMS  candidates  for  the  SDT.  Only  those  DBMSs 
that  ranked  very  high  (fulfilled  all  12  SDT  requirements) 
were  selected  for  further  analysis.  These  DBMSs  are  marked 
with  an  asterisk  (*).  The  seven  DBMSs  that  were  selected 
for  further  analysis  were  ADABAS,  DATACOM/DB,  IDS-I/II  ( DM- 
IV),  IDMS,  MODEL  204,  RAMIS  II  and  SEED.  Each  of  these  DBMSs 
fulfilled  the  12  SDT  requirements. 

At  this  point,  an  attempt  was  made  to  identify  which 
mainframe  of  microcomputer  DBMS  would  be  available  to  a  wide 
range  of  ETES  users  --  the  intent  being  to  select  the  data 
base  from  the  seven  finalists  which  would  run  on  most  of  the 
hardware  owned  by  ETES  users.  This  attempt  to  identify  a 
common  mainframe  or  minicomputer  for  ETES  users  was  not 
fruitful  —  a  variety  of  different  machines  were  used  across 
user  sites.  Many  users  did  not  have  mainframe  or 

minicomputer  nor  did  they  have  access  to  one.  Furthermore, 
since  the  cost  of  obtaining  these  machines  was  high,  it 
appeared  unlikely  that  they  could  obtain  one  in  the  future. 


3. 2. 2. 2  Review  of  Microcomputer  Based  Data  Base  Management 
Systems 

As  the  limitations  of  the  mainframe  or  mini  based  DBMSs 
became  evident,  alternative  configurations  for  the  SDT  data 
base  were  examined.  _One  configuration  whjch  was  identified 
as  having  considerable  merit  was  the  distributed  processing 
configuration.  In  the  distributed  processing  configuration, 
a  centralized  data  base  for  each  weapon  system  (or  weapon 
system  alternative)  would  be  stored  on  a  mainframe 
computer.  At  periodic  intervals,  users  would  transfer  a 
copy  of  the  data  base  from  the  mainframe  to  a  local 
microcomputer.  Once  on  the  micro,  users  would  perform 
standard  data  base  management  functions  (input,  output, 
modify).  Thus,  all  major  data  base  management  functions 
could  be  performed  independently  on  the  microcomputer.  Once 
users  have  completed  their  activities  with  the  data  base, 
they  could  transfer  the  updated  version  to  the  mainframe.  A 
detailed  audit  trail  would  be  kept  for  each  weapon  system  so 
that  users  can  systematically  track  and  assess  system 
changes.  The  distributed  architecture  has  several  important 
advantages  over  other  configurations.  Specifically,  it  (1) 
minimizes  computer  resource  requirements  (the  average  user 
is  only  required  to  purchase  the  microcomputer),  (2) 
minimizes  on-line  computer  charges,  (3)  allows  users  the 
flexibility  of  conducting  their  own  independent  analyses, 
and  (4)  provides  capabilities  for  the  maintenance  of  a 
centralized  data  base,  thus  maintaining  data  integrity. 

After  discussions  with  the  COTR,  it  was  decided  that  the  SDT 
would  be  designed  to  meat  the  requirements_for  a  distributed 
processing  architecture  and  the  search  began  for  a 
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mi G recompute r- based  data  base  management  system  which  would 
support  this  configuration. 


Several  of  the  most  popular  and  widely  used  microcomputer 

based  data  base  management  systems  (for  example,  dBASE  II, 

MDBS  III)  were  examined  to  determine  if  any  of  these  data 
bases  could  meet  the  ETES  requirements  identified  in  Section 
2  and  3.1.  This  review  indicated  there  were  two  major 
problems  with  existing  off-the-shelf  microcomputer  based 
data  base  management  systems  which  limited  their 
applicability  to  ETES.  First,  all  of  the  micro-computer 
based  data  base  management  systems  used  a  query  language  to 

access  the  data  and  obtain  output.  It  was  felt  that  these 

query  languages,  while  considerably  simpler  than  high-level 
languages,  were  too  complicated  for  the  potential  ETES  user 
to  learn  and/or  use.  Second,  it  was  felt  that  these 
generalized  data  bases  would  not  take  full  advantage  of  the 
common  data  elements  which  are  used  over  and  over  again  in 
the  development  of  each  weapon  system  (One  of  the  major 
objectives  of  ETES  was  to  identify  a  common  set  of  data 
elements  which  could  be  used  across  systems).  In  this 
sense,  the  flexibility  of  these  data  bases,  which  would  be 
an  advantage  in  most  applications,  was  considered  to  be  a 
disadvantage  for  the  ETES  application.  Recognizing  these 
limitations,  it  was  decided  to  build  a  specialized 

microcomputer-based  data  base  management  system  which  was 
(1)  tailored  to  meet  the  specific  needs  of  early  task 

training  estimation  for  Army  weapon  system  and  (2)  used  more 
"user-friendly"  human  computer  dialogue  techniques  such  as 
menu  selection  to  access  and  output  data. 


3. 2. 2. 3  Selection  of  SDT  Hardware/Software 


I 

i- 

I  Once  it  was  determined  that  a  tailored  microcomputer-based 

I  management  system  would  be  used  for  the  SDT,  an 

!  investigation  was  conducted  to  identify  the  microcomputer 

I  and  computer  languages  which  would  be  used  for  SDT 

development.  As  part  of  this  investigation,  an  attempt  was 
made  to  identify  the  brand  of  microcomputer  that  was  most 
popular  among  ETES  users  in  the  Army.^  This  investigation 
j  indicated  that  the  Apple  computers  were  most  popular  among 

potential  ETES  users.  For  example,  Apple  computers  were 
being  used  by  ARI  and  by  several  of  the  TRADOC  schools. 

I  Once  it  was  decided  that  an  Apple  was  to  be  used,  the 

specific  hardware  configuration  that  was  to  be  used  to 
develop  the  SDT  was  identified.  This  configuration  is 
listed  in  Table  3-3.  The  Apple  III,  rather  than  the  Apple 
II,  was  selected  as  the  microcomputer  to  be  employed  in  the 
SDT  because  the  Apple  III  had  the  capability  to  be  used  with 
a  external  hard  disk  system.  Such  a  system  could  provide 
the  memory  needed  for  the  SDT  data  base  management 
functions^ 

To  provide  a  capability  for  testing  the  distributed 
processing  capability  of  the  SDT,  it  was  decided  to  use  the 
contractors  (DRC's)  mainframe  computer  (the  Honeywell  DPS 


^  A  more  formal  attempt  was  made  to  identify  any  Army 
wide  plans  to  purchase  a  single  brand  of  microcomputer. 
This  investigation  did  not  produce  any  fruitful  results. 

^  Since  the  SDT  was  developed,  Apple  has  developed  a 
version  of  the  Apple  II,  the  Apple  I  IE  which  can  be  used 
with  the  external  hard  disk. 
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Table  3-3 

SDT  Hardware  Configuration. 

MICROCOMPUTER 

CONFIGURATION 

MAINFRAME 

APPLE  III 

DPS  8/52 

MODEM 

PRINTER 

MONITOR 

FLOPPY  DISK  DRIVES 

•PROFILE  MASS  STORAGE  (HARD  DISK) 

OR 

ADDITIONAL  FLOPPY  DISK  DRIVES 

8/3  2)  since  it  provided  a  low  cost  mechanism  for 
demonstrating  this  capability.  Because  of  resource 

limitations,  only  a  very  limited  distributed  processing 
capability  was  to  be  demonstrated  during  the  ETES 
development  contract.  It  was  decided  that  the  full 
distributed  processing  capability  would  either  be  achieved 
in  the  Man  Integrated  System  (MIST)  contract,  which  is 
currently  integrating  ETES  with  several  other  ARI  MPT 
projects,  or  in  the  ETES  implementation  contract  which  was 
to  follow  the  ETES  development  contract. 

After  the  hardware  configuration  was  identified,  the 
computer  language  to  be  used  for  the  SDT  software  was 
identified.  PASCAL  was  selected  as  the  language  for 
developing  the  SDT  and  other  ETES  automated  tools  because  of 
(1)  its  widespread  usage  on  both  microcomputers  and 
mainframes,  and  (2)  its  highly  structured  nature  which 
facilitates  modular  program  construction  and 

transportabi ii ty. 


Because  of  resource 


3.2. 2.4  Past  Efforts  In  Develqping  System-Specific  Data 
Bases 

Concurrent  with  the  examination  of  data  base  management 
technologies,  past  efforts  in  developing  system-specific 
human  resource  data  bases  were  examined.  Table  3-4  lists 
the  major  data  bases  that  were  examined  during  this 


process. 


More  details  on  each  of  these  data  bases  are 


provided  in  the  section  which  follow. 


Table  3-4 

PAST  EFFORTS  AT  HUMAN  RESOURCE  DATA  BASE  DEVELOPMENT* 

(1)  Logistics  Support  Analysis  Record  (LSAR) 

(2)  Unified  Data  Base  of  Air  Force  Human  Resource  Lab 

(3)  Consolidated  Data  Base  (CDB)  of  Navy/Army  HARDMAN  Projects 

(4)  Structured  Approach  to  Training  (SAT)  Program  for  the  B1 -Bomber 

(5)  Navy  Enlisted  Professional  Information  Support  System  (NEPDISS) 

*  Efforts  are  listed  in  terms  of  their  decreasing  relevance  to  the  ETES  SDT. 


o  Logistics  Support  Analysis  Record 

One  major  effort  which  is  closely  related  to  the  objectives 
and  goals  of  the  SDT  is  the  Logistics  Support  Analysis 
Recore  (LSAR).  MIL-STD-1388  states  that  the  goal  of  the 
LSAR  is  to  be  the  "single  source  of  validated,  integrated, 
design-related  logistic  data  pertinent  to  the  acquisition 
program. " 

Table  3-5  lists  the  systan  elements  that  are  described  by 
the  LSAR  and  the  major  weaknesses  of  the  current  LSAR  in 
respect  to  the  goals  and  objectives  of  the  SDT.  As  Table  3- 
4  indicates,  the  LSAR  has  several  weaknesses  which  limit  its 
use  as  a  comprehensive  system  description  technology  for 
human  resource  assessment. 

First,  there  are  several  important  system  elements  (e.g., 

system  functional  requirements,  collective  tasks)  which  the 
LSAR  does  not  describe.  Failure  to  describe  the  system 
functional  requirements  is  particularly  distressing,  since 
these  functional  requirements  provide  the  foundation  on 

which  all  other  system  elements  depend.  Lack  of  a 

systanatic  description  of  functional  requirements  makes  it 
extremely  difficult  for  training  developers  and  others  who 
are  tasked  with  relating  their  particular  system  elements  to 
overall  mission  performance  and  its  associated  functions. 
For  instance,  it  makes  it  extremely  difficult  to  relate 
human  tasks  to  mission  performance.  Given  its  lack  of  a 
capability  for  describing  system  functional  requirements  or 
projected  system  elements,  it  is  not  surprising  that  the 
LSAR  is  currently  not  applied  during  the  concept  exploration 
phase  of  the  acquisition  process  and  seldom,  if  ever, 

applied  during  the  validation  and  demonstration  phase. 


Table  3-5 

OVERVIEW  OF  LSAR  AND  ITS  MAJOR  WEAKNESSES 


System  Elements  Described  by  LSAR 

•  Equipment  (work  breakdown  structure,  work  unit  code,  nomenclature, 
reliability,  maintainability,  failure  symptoms,  failure  effect  and  criticality, 
maintenance  concept) 

•  Tasks  (task  code,  frequency,  elapsed  time,  skill  specialty,  man  hours, 
requirements  for  training  equipment,  support  equipment,  tools,  task  elements, 
aggregate  maintenance  man-hour  requirements) 

•  Support  and  Test  Equipment  (physical  characteristics,  associated  equipment, 
associated  tasks,  associated  training,  special  skill  requirements) 

•  Facilities  (associated  equipment  and  tasks,  general  requirements,  lead 
times,  type  of  construction,  utilities,  facility  unit  cost) 

•  Skills  (associated  task  and  equipments,  specialty  codes,  aptitude,  rank/rate, 
special  physical  and  mental  requirements,  educational  requirements, 
additional  training  requirements) 

•  Supply  Support  (part  no.  and  nomenclature,  physical  description,  associated 
equipment,  allowance  quantity,  distribution) 


Major  Weaknesses  of  LSAR* 

.  Does  not  describe  system  functional  requirements 
.  Does  not  provide  adequate  description  of  operator  tasks 

•  Does  not  describe  task  characteristics  or  performance  information 

•  Does  not  describe  collective  tasks 

•  Does  not  adequately  describe  skills 

•  Does  not  adequately  describe  training  program  elements 

•  Does  not  provide  mechanism  for  describing  estimated  or  projected  elements 

•  Is  not  applied  in  early  phases 

«  Does  not  have  data  base  management  capability 

•  Cannot  generate  tasks  or  other  input  data 


*Many  of  these  limitations  are  apparently  being  dealt  within  the  present  LSAR 
improvement  programs. 


Hence,  its  value  as  a  data  base  to  support  early  human 
resource  assessment  is  very  minimal  indeed. 


I 

I 

Second,  there  are  a  number  of  other  systems  elements  which 
are  described  by  the  LSAR  but  are  not  described  adequately 
or  in  enough  detail  (e.g.,  operator  tasks,  task 
characteristics,  training  program  elements  skills).  The 
emphasis  of  the  LSAR  on  maintenance  and  maintenance  tasks  is 
quite  obvious.  This  emphasis  makes  it  extremely  difficult 
to  develop  or  maintain  adequate  descriptions  of  operator 
tasks.  For  all  types  of  tasks,  the  LSAR  does  not  fully 
descrioe  the  task  characteristics  and  performance 
information  that  is  needed  by  training  and/or  human  factors 
specialists  to  adequately  assess  their  components  of  the 
system.  The  training  portion  of  the  LSAR  places  an  emphasis 
on  training  equipment  and  devices  and  ignores  other 
important  aspects  of  the  training  program  (e.g.,  course 
modules ) , 

Third,  at  a  more  conceptual  level,  the  LSAR  does  not  provide 
an  adequate  capability  for  describing  estimated  or  projected 
system  elements.  Such  estimates  are  necessary  during  the 
early  phases  of  the  acquisition  process. 

Fourth,  the  LSAR  was  not  conceived  as  an  automated  data  base 
management  system  description  —  that  is,  as  an  automated 
system  for  describing,  updating,  and  expanding  system 
concepts  and  communicating  this  information  to  system 
users.  It  should  be  noted  that  the  Army,  through  the 
DARCOM  Materiel  Readiness  Support  Activity,  has  been  a 
leader  in  "automating  the  LSAR."  However,  this  automation 
apparently  refers  only  to  the  use  of  computerized  algorithms 
for  aggregating  certain  LSAR  elements  or  for  presenting 
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printed  outputs  of  reports.  It  is  not  designed  to  be  an 
interactive  system.  More  important,  the  automated  LSAR  does 
not  provide  for  the  automated  description  of  system 
concepts,  updates,  changes  and  expansions  through  a 
comprehensive  data  base  management  system.  This  is  due  to 
the  fact  that  the  LSAR  does  not  have  a  systematic  internal 
structure  linking  the  various  system  elements  to  one 
another. 

o  Air  Force  Human  Resource  Lab  Unified  Data  Base 

The  Air  Force  Human  Resource  Lab  (AFHRL)  has  initiated  a 
program  to  develop  a  Unified  Data  Base  (UDB).  The  goals  of 
the  UDB  are  very  similar  to  the  SDT  (see  Thomas,  Newhouse 
and  Hankins,  1980;  Thomas  and  Hankins,  1980).  Ultimately, 
the  UDB  is  designed  to  provide  "a  centrally  located  data 
base  of  human  resource  related  information  for  utilization 
in  the  weapon  system  acquisition  process  to  influence 
hardware  concepts  and  design."  The  UDB  is  to  be  supported 
by  a  Data  Generating  Technology  Data  Base  (DGTB)  which  is 
intended  "to  generate  generic  data  to  fill  in  the  needs  of 
users  where  the  data  systems,  and  likewise  the  UDB,  would 
leave  voids."  Thus,  the  DGTB  is  somewhat  similar  to  the 
ETES  training  estimation  aids  and  procedures. 

At  the  time  the  ETES  review  was  conducted,  UDB  development 
efforts  had  focused  on  (1)  an  assessment  of  existing 
historical  data  bases  which  would  feed  the  UDB,  particularly 
the  projected  portions  of  the  UDB,  (2)  a  description  of  the 
weapon  system  design  process  with  respect  to  the  potential 
use  of  the  UDB,  (3)  an  assessment  of  user  needs  in  terms  of 


adequacy  of  current  technology  and  data4,  and  (4)  the 
development  of  a  plan  for  (JDB/DGTB  development. 

At  the  time  the  ETES  review  was  conducted,  a  description  of 
the  actual  data  elements  to  be  included  in  the  UDB  was  not 
available  (this  was  to  be  developed  in  future  phases  of  the 
study).  However,  by  examining  the  types  of  historical  data 
bases  which  are  projected  to  be  used  by  the  UDB,  it  was 
possible  to  make  some  estimates  of  what  it  would  contain  and 
to  assess  some  of  its  potential  limitations. 

These  limitations  point  out  the  differences  between  the  UDB 
and  the  ETES  SDT.  These  differences  are  actually  quite 
significant  despite  the  similarity  in  the  goals  of  these  two 
systems  (see  Table  3-6). 

The  first  limitation  of  the  UDB  is  its  emphasis  on 
maintenance  tasks  and  personnel.  The  UDB,  like  the  Air 
Force  Coordinated  Human  Resources  Technology,  emphasizes 
maintenance  behavior  and  the  use  of  historical  data  bases 
related  to  maintenance.  There  is  little  relevant  discussion 
of  the  procedures  and  mechanisms  for  developing  or 
describing  operator  tasks  or  training  requirements. 

This  emphasis  on  maintenance  tasks  is  closely  related  to  a 
second  limitation  of  the  UDB;  namely,  its  emphasis  on 
aircraft  systems  and  on  Air  Force  data  bases.  In  the  Air 


In  the  examination  of  the  utilization  of  human  resource 
data  in  tradeoffs,  it  is  interesting  to  rv^te  that  lack  of 
information  and  lack  of  appropriate  analytical  tools  were 
seen  as  two  of  the  major  types  of  limitations  on  the  use  of 
human  resource  assessment. 


Table  3-6 

LIMITATIONS  OF  t'HE  UDB 


Focuses  almost  exclusively  on  maintenance  tasks. 

Emphasizes  aircraft  systems 

Does  not  appear  to  adequately  describe  functional  requirements, 
collective  or  team  tasks,  task  characteristic  or  performance  data, 
and  training  program  elements 

Is  not  based  upon  comprehensive  data  base  management  syster  n 
structure 

Is  geared  for  use  by  sophisticated  users 


Cannot  generate  tasks  and  other  input  data 


Force,  the  role  of  enlisted  operators  is  much  less 
significant  than  it  is  in  the  Army  or  Navy.  Hence,  it  is 
not  surprising  that  the  UDB  has  focused  on  the  maintenance 
of  aircraft  systems. 

Third,  there  are  number  of  other  system  elements  which  the 
UDB  would  appear,  at  least  at  the  present  time,  not  to 
describe.  These  elements  include  functional  requirements, 
collective  or  team  tasks,  task  characteristics,  and 
performance  data  suitable  for  training  and  human  factors 
analytical  activities,  and  training  program  elements.  (This 
failure  to  describe  certain  elements  would  not  be  critical 
if  the  UDB  had  the  proper  data  base  management  structure  to 
handle  additional  system  elements.  Unfortunately,  it  appears 
that  it  does  not  have  this  capability). 

Fourth,  and  perhaps  most  important,  the  UDB  again  does  not 
appear  to  be  based  upon  a  data  base  structure — that  is,  a 
structure  which  represents  the  implicit  relationships  among 
the  various  system  elements.  Such  a  data  base  management 
structure  would  provide  a  mechanism  for  describing  the  basic 
structure  of  a  developing  system  which  was  independent  of 
the  various  user  viewpoints  of  the  data.  This  data 
independence  would  increase  the  capability  for  relating 
various  descriptions  of  the  system  to  one  another,  for 
updating  and  refining  the  data,  and  for  adding  new  elements 
to  the  data  base  in  a  systematic  modular  fashion  with 
minimum  destruction  of  existing  programming  thus  providing 
the  basis  for  a  true  data  base  management  capability. 


Fifth,  the  UDB  ap^jears  to  be  geared  for  use  by  technical 
personnel  who  have  sophisticated  analytical  and/or  computer 
programming  exper ienc e--un 1 i ke  the  SDT  which  is  geared  for 


use  by  personnel  with  little  background  in  computers. 
Because  of  this  difference  in  emphasis/  it  is  not  surprising 
that  the  UDB  does  not  specify  or  deal  with  the  human  factors 
of  human-computer  interactions  as  with  the  SDT,  which  is 
specifically  geared  for  utilization  by  uninitiated  users  and 
will  attempt  to  employ  the  latest  guidelines  on  human- 
computer  interfaces.  Because  of  its  lack  of  consideration 
of  human  factors  issues,  the  UDB  does  not  attempt  to  provide 
procedures  for  assisting  the  user  in  generating  tasks  or 
other  input  data  elements. 

o  Consolidated  Data  Base  (CDB)  of  Hardman 
Methodology 

The  Navy  has  a  program,  called  the  HARDMAN  program  (hardware 
versus  manpower  procurement),  to  develop  a  methodology  to 
systematically  assess  the  manpower,  personnel,  and  training 
requirements  of  emerging  weapon  systems,  with  particular 
emphasis  on  developing  predictions  for  the  early  phases  of 
the  acquisition  process.  The  HARDMAN  methodology  has  been 
applied  to  a  number  of  different  Navy  systems  and  has  been 
modified  for  use  by  the  Army  and  applied  to  several  Army 
weapons  systems.  (See  Appendix  D  for  a  discussion  of 
HARDMAN )  . 

The  application  of  the  HARDMAN  methodology  is  supported  by 
the  development  of  a  system-specific  "data  base"  which  is 
designed  to  contain  all  of  the  inputs  and  outputs  of  each  of 
the  steps  in  the  HARDMAN  methodology  and  provide  an  audit 
trail  for  monitoring  the  data  elements  which  are  developed. 
Table  3-7  lists  the  data  elements  described  by  the  CDB.  Like 
the  other  current  human  resource  data  bases,  the  CDB 
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Table  3-7 

DATA  ELEMENTS  CONTAINED  IN  CDB 


General  System 

•  Requirements  Documents 

.  Study  Plans  and  Objectives 

.  Technology  Base  Studies 

•  Projected  Operational  Environment 

.  System  Functions  and  Performance  Requirements 

.  Program  Constraints 

.  Minimal  Essential  Elements  of  Information  List 

«  Audit  Trail  Files 

.  Worksheets 

.  CDB  Index 

.  Predecessor  Equipment  List  and  Related  Data 

•  Reference  Equipment  List  and  Related  Data 
.  Predecessor  and  Reference  Reliability  Data 

Manpower* 

.  Workload  Taxonomy 

•  Indirect  Workload  Factors 

.  Task  Event  Networks 

•  Manpower  Model  Data 

•  Manpower  Metrics  and  Associated  Values 

.  System  Manning  (MOS,  Skill  Level,  Duty  Positions) 

Training* 

•  Task  and  Skill  Data 

•  Course  Catalogue 

.  Course  Outlines 

.  Course  Methods/Media 

.  Course  Costing  Data 

.  Course  Scenario  Information 

«  Career  Path  Information 

.  Training  Concept 

.  Training  Device  and  Equipment 

.  Steady  State  Resource  Requirements 

.  Steady  State  Course  Costs 

.  Replacement  Personnel  Requirements 

.  Task  Selection  and  Assignment  Algorithms 

.  Facilities  Requirements 

Personnel* 

•  Career  Path  Data 

•  Career  Path  Statistics  (Attrition,  Promotion.  Upgrade) 


*A1I  elements  for  predecessor,  reference,  and  baseline  systems  except  where  noted. 


ii 
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has  several  limitations  with  respect  to  the  SDT 
requirements . 

The  major  limitation  of  the  CDB  is  that  only  parts  of  it  are 
automated.  Thus,  it  cannot  provide  a  computerized  data  base 
management  capability.  Another  major  limitation  of  the  CDB 
is  that,  like  the  UDB,  it  does  not  contain  a  systematic 
scheme  for  relating  the  various  system  elements  to  one 
another,  a  scheme  which  would  be  independent  of  specific 
input  and  output  requirements.  Thus,  the  CDB  is  not  really 
a  true  data  base  management  system  since  it  does  not  have  an 
automated  capability  for  linking  various  system  elements  to 
another  for  retrieving  data  elements. 

Finally,  the  CDB  does  not  provide  any  extensive  automated 
capabilities  for  generating  input  data  formats  or  actual 
input  data  elements. 

o  SAT  Program  for  the  B-1  Bomber 

The  Structural  Approach  to  Training  (SAT)  program  for  the  B- 
1  bomber  represents  a  relatively  early  attempt  to  develop  a 
system-specific  data  base  to  support  instructional  systems 
development  (see  Sugarraan,  Johnson,  and  Ring,  1975). 

The  SAT  consisted  of  two  major  elements,  a  data  base  (the 
contents  of  which  are  displayed  in  Table  3-8)  and  two 
computerized  aids.  One  aid  is  a  sorting  model  for  the 
storage,  retrieval,  collating,  and  updating  of  mission/ 
function  task  analyses  and  supporting  data;  the  other  is  an 
analytical  model  for  providing  cost  and  training  estimates 
of  the  B-1  bomber  training  system. 


Table  3-8 

SAT  DATA  ELEMENTS  AND  LIMITATIONS 


System  Elements  Descrilsd  by  SAT  Data  Base 

•  Tasks  (title,  task  element  number,  operator  behavior,  task  duration, 
crew  interaction,  previous  task  element,  task  characteristics,  and 
performance  data) 

«  Control/display  information  (associated  system,  synonyms) 

•  Behavioral  objectives  (title,  initial  conditions,  concurrent  behaviors, 
performance  criteria,  enabling  and  ancillary  objectives,  operators, 
interactions,  task  elements,  objective  criticality,  objective  difficulty) 

Limitations  of  SAT  Data  Base 

Is  geared  for  one  specific  system 

Is  not  designed  to  provide  generic  data  base  management  capability 

Does  not  systematically  describe  system  functional  requirements  and 
design  concepts 

Does  not  include  training  program  elements  in  automated  portion 
of  the  data  base 

Is  geared  for  sophisticated  users 

Cannot  generate  tasks  and  other  input  data 


The  SAT  data  base  is  interesting  in  that  it  is  probably  the 
only  past  effort  which  has  attempted  to  (1)  systematically 
describe  task  characteristics  in  a  format  which  is  amenable 
to  the  application  of  automated  training  aids  for 
determining  the  tasks  to  be  trained  and  selecting  methods 
and  media,  and  (2)  systematically  describe  the  task 
performance  characteristics  of  equipment  (e.g.,  relation¬ 
ships  of  tasks  to  controls  and  displays).  Such  task 
performance  data  is  critical  to  human  task  performance 
simulation  models. 

However,  despite  its  desirable  features,  the  SAT  data  base 
also  has  several  limitations  which  restrict  its 
applicability  to  the  SDT.  First,  the  SAT  data  base  elements 
and  programs  were  specifically  designed  to  fit  one  system-- 
the  B-1  bomber.  Thus,  all  of  its  task  and  control/display 
dictionaries  and  structures  are  only  applicable  to  that 
system.  The  SAT  was  not  designed  to  be  a  generic  data  base 
system  which  could  be  applied  across  a  wide  range  of  weapon 
sys  t«ns. 

Second,  the  SAT  does  not  describe  several  important  system 
elements  such  as  functional  requirements  and  design/hardware 
e  leraents. 

Third,  training  program  elements  are  described  but  not 
included  in  the  automated  data  base. 

Fourth,  the  SAT  is  geared  for  very  sophisticated  users  with 
extensive  computer  experience. 

Fifth,  the  SAT  is  not  structured  to  assist  users  in 
developing  input  data  formats  or  actual  input  data  elements 
such  as  tasks. 
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Navy  Enlisted  Professional  Development  Infonnation 
Support  System  (NEPDISS) 


< 


The  objectives  of  the  NEPDISS  are  more  limited  than  the 
goals  of  the  ether  human  resource  data  bases  described 
above.  The  NEPDISS  is  specifically  designed  to  store  and 
retrieve  data  related  to  training  program  development  (see 
Davis,  19  77  for  a  description).  Thus,  it  is  primarily 
designed  to  describe  task  and  training  data  (see  Table  3- 
9).  The  only  description  of  equipment-related  concepts  in 
NEPDISS  is  in  the  task  statements  of  the  task  portion  of  the 
data  base.  Other  major  limitations  of  the  NEPDISS  are  its 
lack  of  capability  for  describing  projected  system  elements, 
its  apparent  lack  of  appropriateness  for  use  by  uninitiated 
users,  its  lack  of  a  capability  for  generating  tasks  and 
other  data  impacts,  and  most  important,  its  lack  of  a  true 
data  base  management  capability  for  updating  and  refining 
system  elements. 

Despite-  the  weaknesses,  it  is  important  to  note  that  the 
NEPDISS  is  especially  strong  in  describing  task  and  skill 
related  requirements  which  are  appropriate  for  training  and 
personnel  analysis. 

o  Other  Data  Bases 

There  are  a  number  of  other  data  bases  which  attempt  to  deal 
with  some  of  the  issues  related  to  the  SDT.  For  instance, 
the  Consolidated  Occupational  Data  Analysis  Program  (COCAP) 
and  the  Training  Developments  Information  System  (TDIS)  are 
Army  data  bases  which  also  deal  with  task  description.  The 
CODAP  focusses  on  tasks  from  the  perspective  at  a  single  MOS 
while  the  TDIS  focusses  on  canmon  tasks  which  are  applicable 
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Table  3-9 


NEPDISS  DATA  LIMITATIONS 

Does  not  describe  system  functional  requirements,  design  concepts,  training 
program  elements  or  collective  tasks. 

Is  geared  for  use  by  sophisticated  users. 

Cannot  generate  task  and  ether  input  data. 

Is  not  designed  to  describe  projected  system  elements. 

Does  not  provide  comprehensive  data  base  management  capability  for  updating 
and  refining  system  elements. 


across  MOS.  However,  neither  one  is  geared  for  use  in 
describing  the  hardware/software  design,  tasks,  and  training 
characteristics  of  a  emerging  weapon  systems. 


3.3  REVIEW  RESEARCH  RELATED  TO  HUMAN  COMPUTER  INTERACTIONS 

One  of  the  major  problems  surrounding  the  weapon  system 
design  process  is  the  communication  and  flow  of  information 
among  the  participants  in  this  more  general  process.  The 
ETES  SDT  is  specifically  designed  to  deal  with  these 
communication  problems  by  providing  a  centralized,  automated 
data  base  for  describing  and  updating  emerging  system 
concepts  and  providing  direct  access  to  this  data  base  to 
all  participants  in  the  weapon  system  design  process.  Thus, 
the  SDT  provides  a  systematic  vehicle  through  which  the 
participants  in  the  weapon  system  design  may  communicate. 
To  provide  a  foundation  for  SDT  development  recent  research 
related  to  human-computer  interaction  was  examined.  This 
section  describes  the  results  of  that  examination. 


The  section  is  divided  into  two  subsections.  The  first 
describes  the  SDT  requirements  related  to  human  computer 
interactions.  The  second  subsection  reviews  psychological 
research  relating  to  the  process  of  human  computer 
interactions  and  summarizes  the  implications  that  ♦•his 
literature  has  for  the  SDT. 


3.3.1  SDT  Requirements  Related  To  Human  Computer 
Interactions 

Task  1.0  examined  the  problems  in  information  flow  and 
communication  which  limited  early  training  estimation.  This 
task  indicated  that  training  developers  and  other 
participants  in  the  acquisition  process  are  not  system¬ 
atically  receiving  information  on  early  system  concepts  and 
are  not  being  kept  abreast  of  system  changes  and  updates  in 
a  timely  and  systematic  fashion.  This  lack  of  systematic 
communication  makes  it  difficult  if  not  impossible  to 
effectively  assess  training  and  other  human  resources,  and 
this  lack  of  assessment,  in  turn,  makes  it  difficult  to 
effectively  manage  and  control  these  resources. 


In  order  for  the  SDT  to  fill  these  communications 
deficiencies,  the  SDT  must  itself  be  designed  to  facilitate 
easy  and  rapid  communication  with  the  personnel  who  will  use 
It.  Task  1  indicated  that  the  primary  users  of  the  SDT  will 
be  personnel  from  the  staff  of  the  training  developers, 
combat  developers,  and  material  developers.  These  personnel 
are  likely  to  have  had  little,  if  any,  experience  in 
utilizing  computers  or  computerized  data  bases.  Interviews 
with  current  personnel  in  these  organizations  indicated  that 
there  is  also  likley  to  be  very  little  time  or  resources  to 
train  these  personnel  on  ETES-related  activities.  Thus,  it 
was  imperative  that  the  SDT  be  designed  to  (1)  be  utilized 
by  uninitiated  users  who  have  no  background  in  the  use  of 
computers  and  computer  languages  and  (2)  have  minimal 
training  requirements.  Fortunately,  in  recent  years  there 
has  been  a  growing  body  of  literature  on  human-computer 
interactions  and  the  types  of  interactions  which  are 


appropriate  for  uninitiated  minimally  trained  users.  This 
literature  is  reviewed  in  the  subsections  which  follow. 

Before  discussing  this  literature,  it  is  important  to  point 
out  that  the  systematic  study  of  human-computer  interactions 
is  a  relatively  new  area  of  research.  Consequently,  many  of 
the  guidelines  discussed  in  the  next  section  are  only 
rational  schemes  for  dealing  with  human-canputer 
interactions  —  empirical  research  to  support  these 
guidelines  is  generally  not  available.  However,  the 
conceptual  schemes  which  have  been  developed  do  appear  to 
have  a  high  degree  of  face  validity. 

3.3.2  Human-Data  Base  Interaction  Literature  Review 

There  have  been  four  major  efforts  to  survey  and  categorize 
literature  relating  to  human-data  base  interact  ions . 5  More 
details  on  these  four  efforts  is  presented  in  the 
subsections  which  follow. 

3.3.2.  1  Martin's  Work  on  Interactive  Dialogues 

The  first  comprehensive  work  in  human-computer  interactions 
was  conducted  by  Martin  (  19  73)  who  documented  his  work  in  a 
book  on  human-computer  dialogues.  Martin's  book  was  the 
first  attempt  to  provide  a  systematic  set  of  guidelines  or 
human  computer  interactions.  Earlier  work  had  focused  on 


^  This  research  area  is  also  described  as  the  "man- 
machine  interface"  (MMI)  as  well  as  the  human-computer 
interface  or  interaction. 


the  development  of  guidelines  for  conputer  input  devices  or 
output  devices  or  computer  programming  practices  but  had  not 
systematically  covered  dialogue  or  process-related 
questions. 

Martin's  basic  approach  toward  conceptualizing  the  human- 
computer  interactions  was  to  divide  human-computer  inter¬ 
actions  into  18  basic  dialogue  types  and  to  outline  the 
advantages  and  disadvantages  of  each  type  in  terras  of  types 
of  users  and  information  characteristics .  6  Table  3-10 
displays  Martin's  dialogues  types  and  the  estimated 
applicability  to  the  SDT  based  upon  Martin  description  of 
their  advantages  and  disadvantages.  As  Table  3-10 

indicates,  the  most  likely  dialogues  types  for  inclusion  in 
the  SDT  are  menu  selection  dialogues,  form-filling,  and 
question  and  answer  dialogues.  These  were  the  dialogues 
which  Martin  indicated  were  (1)  most  appropriate  for 
uninitiated  users  and  (2)  would  not  require  extensive 
development  costs  or  special  terminals. 

Martin  also  provides  a  series  of  guidelines  to  consider  in 
selecting  input  and  output  devices.  Table  3-11  lists  the 
input  and  output  devices  covered  by  Martin.  To  reduce 
implementation  costs,  it  was  decided  that  the  SDT  would 
utilize  existing  hardware  (see  Section  3. 2. 2. 3  for  a 
description  of  the  SDT  configuration).  This  requires  that 
the  SDT  interactive  input  device  mechanisms  be  restricted  to 
a  keyboard  and  SDT  output  devices  mechanisms  be  restricted 
to  a  printer  and  a  CRT  screen. 


”  Martin  presents  little  empirical  evidence  to  support 
his  concepts  (very  Little  work  has  been  done  in  this 
area).  However,  his  concepts  appear  lo<jical  and  seem  to 
have  3  high  degree  of  "face  validity". 


Table  3-10 

MARTIN'S  DIALOGUE  TYPES  AND  THEIR  APPLICABILITY  TO  SDT* 


1.  Programming  languages 

2.  English-language  dialogue 

3.  Limit  English  input 

*4.  Question  and  answer  dialogues  (in  which  the  computer  asks  the  operator 
a  series  of  questions) 

5.  Dialogue  using  mnemonics 

6.  Dialogue  with  programming-like  statements 

7.  Computer-initiated  dialogues  (in  which  the  operator  responds  to  the  computer 
rather  than  the  computer  responding  to  the  operator) 

8.  Form-filling  (in  which  the  operator  fills  out  a  "form"  on  a  visual  display) 

9.  Menu-selection  dialogues 

10.  Build  dialogue  features  into  special  terminal  hardware 

11.  Dialogues  with  a  light  pen  for  input  (or  other  means  of  pointing  to  the 
screen ) 

12.  Fixed-panel  responses  (in  which  the  computer  responds  with  one  of  a  standard 
set  of  panels) 

13.  Modifiable-panel  dialogues  (in  which  the  panels  can  be  modified  by  the 
programs) 

14.  Graphics  using  chart  displays 

15.  Graphics  using  symbol  manipulation 

16.  Dialogues  with  photographic  frames 

17.  Voice  answerback  dialogues 

18.  Dialogue  via  a  third  party 


*ltems  applicable  to  SDT. 


Table  3-11 


MARTIN'S  CATEGORIZATION  OF  INPUT-OUTPUT  DEVICES 
AND  THEIR  APPLICABILITY  TO  THE  SDT 


Input 

Keyboard 
Lever  set  or 
Rotary  switches 
Push  buttons 

Light  pen  for  point  at  screen 
Finger  pointing  at  screen 
Stylus  for  drawing 
Plate  reader 
Badge  reader 


Output 

♦Typewriter  or  printer 
Alphanumeric  screen 
♦Graphics  screen 
Screen  displaying  film  frames 
Light  panel 
Graph  plotter 
Dials 

Voice  answerback 
Facsimile  machine 


Applicable  to  SDT. 


3. 3. 2. 2  Ramsey  and  Atwood's  Work  on  Human  Factors  in 
Computer  Systems 

Another  major  effort  in  systematically  assessing  human- 
computer  interactions  was  directed  by  H.  Rudy  Ramsey  and 
Michael  Atwood  in  work  sponsored  by  the  Engineering 
Psychology  Programs  of  the  Office  of  Naval  Research.  Ramsey 
and  Atwood  (19  79)  have  developed  a  conceptual  scheme  for 
classifying  different  areas  of  research  relating  to  human- 
cctnputer  interactions.  Table  3-12  presents  this  scheme  and 
also  indicates  which  Atwood  and  Ramsey  categories  are  most 
relevant  to  the  SDT  development.  It  is  important  to  point 
out  that  input  and  output  device  questions  were  less 
relevant  to  the  SDT  because  the  SDT  utilizes  existing  Apple 
III  input/  output  devices. 

In  discussing  user  characteristics,  Ramsey  and  Atwood  review 
several  past  articles  which  have  dealt  with  the  requirements 
and/or  capabilities  of  uninitiated  users.  (e.g.  Card  et  al , 
1974;  Eason,  et  al  19  7  5;  Evans,  19  7  6;  Martin,  19  73  ; 
Nickerson  and  Pew,  19  71;  and  Thompson,  19  71).  Atwood  and 
Ramsey  indicate  that  interactions  by  these  users  can  be 
facilitated  if  the  computer-initiated  or  natural  language 
dialogues  are  used,  and  they  point  out  that  natural  language 
dialogues  are  very  expensive  to  develop. 

As  was  noted  earlier,  Ramsey  and  Atwood  indicate  that 
ccmputer  initiated  dialogue  would  seem  to  a  much  more 
effective  means  of  communication  with  uninitiated  users,  who 
are  exactly  the  type  of  users  which  will  utilize  the  SDT. 

Ramsey  and  Atwood  rx)int  out  that  compute  r- i  n  i  t  ia  ted  dialogue 
has  several  advances.  First,  this  approach  to  dialogue 


allows  the  systan  to  rely  on  the  passive  vocabulary  of  the 

user  (the  set  of  words  which  the  user  can  recognize  and 

understand),  which  is  typically  much  larger  than  the  user's 
active  vocabulary  (words  which  the  user  can  generate  and  use 
without  prompting).  Second,  it  allows  the  designer  to 
implicitly  convey  to  the  user  a  "mental  model"  of  the 

system's  dialogue  structure.  The  major  disadvantage  of 
computer-initiated  dialogue  is  the  frequent  delay  it  may 

produce  for  experienced  users. 

Ramsey  and  Atwood  also  present  a  scheme  for  classifying 
different  types  of  interactive  dialogues,  and  list  the 
advantages  and  disadvantages  of  each  type.  A  summary  of 
this  discussion  is  presented  in  Table  3-13.  Two  types  of 
dialogue  seem  especially  applicable  to  the  SDT  because  of 
their  emphasis  on  computer-initiated  dialogue  --  form 
filling  and  menu-selection. 

Form-filling  is  often  used  in  situations  in  which  the  user's 
input  is  dominated  by  parameter  values,  rather  than 
commands.  Many  system  attributes  involve  this  type  of 
data.  hence,  form-filling  would  appear  to  be  particularly 
useful  as  a  data  input  mechanism  for  data  attributes. 


Menu  selection  is  described  by  Ramsey  and  Atwood  as  the 
"archetype  of  computer-initiated  dialogue."  Unlike  question 
and  answer  dialogue  or  form-filling,  all  of  the  times  to  be 
selected  appear  on  the  screen,  and  thus  the  user  need  only 
recognize  the  desired  action.  Also,  a  simple  menu-selection 
dialogue  ordinarily  requires  only  one  user  input  (on  the 
keyboard  or  screen),  rather  than,  for  example,  the  series  of 
keystrokes  required  to  type  a  whole  word.  Redsdale  (19  70) 
reports  a  study  which  documented  the  effectiveness  of  menu 


selection  with  naive  users.  The  study  indicated  that  menu 
selection  was  especially  effective  when  used  as  means  of 
obtaining  answers  to  a  set  of  branching  questions. 

It  is  especially  important  to  note  that  menu  selection  is  a 
highly  effective  dialogue  method  for  hierarchical  search 
because  of  its  reliance  on  the  user's  passive  vocabulary  and 
recognition  memory.  Hence,  menu  selection  is  particularly 
applicable  to  information  retrieval  (see  Thompson,  1969, 
1979  for  a  more  extensive  discussion  of  menu  selection  as  a 
data  retrieval  mechanism). 


3. 3. 2.3  Smith's  Work  on  Man-Machine  Interface 

Another  major  effort  related  to  the  assessment  of  human- 
computer  interactions  was  Sidney  Smith's  (1980)  work  on  the 
development  of  guidelines  for  the  man-machine  interface  in 
C3  systems.  This  work  was  sponsored  by  the  Air  Force 
Electronic  Systems  Command. 

Like  Ramsey  and  Atwood,  Smith  (1980)  has  developed  a  scheme 
for  categorizing  topic  areas  related  to  human-computer 
interactions.  Table  3-14  displays  Smith's  schemes  and  the 
categories  of  Smith's  work  which  had  the  most  applicability 
to  the  SDT.  Selected  aspects  of  Smith's  (1980)  major  report 
in  this  area  which  were  relevant  to  ETES  are  reviewed  below. 

o  Dialogue  Types 


Smith,  like  many  other  investigators  in  this  area,  notes 
that  compute r- in i t ia ted  dialogue  types  (e.g.,  form- f i 1 1 i ng , 
menu  selection)  are  more  appropriate  for  uninitiated  users 


Table  3-14 


SMITH’S  SCHEME  FOR  CLASSIFYING  HUMAN-COMPUTER 
INTERACTION  INFORMATION* 


1.0  DIALOGUE  TYPE 

1.1  Question  and  Answer 

1.2  Form  Filling 

1.3  Menu  Selection 

1.4  Function  Keys 

1.5  Command  Language 

1.6  Query  Language 

1.7  Natural  Language 

1.8  Graphic  Interaction 

2.0  DATA  ENTRY/INPUT 

2.1  Position  Designation 

2.2  Direction  Designation 

2.3  Data  Type 

2.4  Entry  Formats 

2.5  Data  Validation 

2.6  Data  Processing 

3.0  DATA  DISPLAY/OUTPUT 

3.1  Data  Type 

3.2  Data  Density 

3.3  Data  Aggregation 

3.4  Data  Coding 

3.5  Display  Partitioning 

3.6  Display  Selection 

3.7  Data  Coverage 

3.8  Display  Update 

3.9  Data  Selection 


4.0  SEQUENCE  CONTROL 

4.1  Transaction  Selection 

4.2  Interrupt 

4.3  Context  Definition 

4.4  Error  Management 

4.5  Alarms 

5.0  USER  GUIDANCE 

5.1  Status  Information 

5.2  Routine  Feedback 

5.3  Error  Feedback 

5.4  Instructional  Aids 

6.0  DATA  TRANSMISSION 
COMMUNICATION 

6.1  Data  Transfer 

6.2  Data  Type 

6.3  Transmission  Control 


Table  derived  from  Smith  tl980J. 


than  are  user-initiated  dialogues  (e.g.,  programming 
languages).  Table  3—15  displays  Smith's  categorization  of 
the  different  dialogue  types  and  his  estimation  of  user 
training  and  response  time  associated  with  eacTT  type.  Based 
upon  Smith's  estimates  question  and  answer,  form-filling  and 
menu  selection  would  seem  to  be  the  most  appropriate 
dialogue  formats  for  the  types  of  uninitiated  users  who  will 

use  the  SDT  (response  time  is  not  a  critical  issue  for  the 
SDT)  . 


o  Data  Entry/Input 

In  discussing  data  entry/input  topics,  Smith  mentions  some 
general  guidelines  that  should  be  considered  in  the 
construction  of  data  entry  mechanisms.  First,  an  operator 
should  seldom  be  required  to  enter  the  same  data  twice  or 
enter  a  data  item  already  entered  by  another  operator.  He 
suggests  that  the  way  to  do  this  is  to  program  the  computer 
to  maintain  context  (that  is,  the  computer  should  be  able  to 
access  all  data  related  to  the  user's  input).  Second, 
conputer  systems  should  be  flexible.  This  means  that  the 

user  should  be  able  to  set  his  own  pace,  cancel  incomplete 
transactions,  etc. 

o  Data  Di splay /Output 

Much  of  the  material  covered  in  Smith's  discussion  on 
display/  output  is  redundant  with  Ramsey  and  Atwood's 
work.  Hence,  it  is  not  repeated  here.  — 
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Table  3-15 

SMITH'S  SCHEME  FOR  CLASSIFYING  DIALOGUE  TYPES* 


Required  System 


Dialogue  Type 

User  Training 

Response 

Question  and  Answer 

Little/None 

Moderate 

Form  Filling 

Moderate/Little 

Slow 

Menu  Selection 

Little/None 

Very  Fast 

Function  Keys  with 

Command  Language 

High/Moderate 

Fast 

User-Initiated  Command 

High 

Fast 

Language 

' 

Query  Languages 

High/Moderate 

Moderate 

Natureil-Language  Dialogues 

Moderate 

(potentially  little) 

Fast 

Interactive  Graphics 

High 

Very  Fast 

Tim 


*Table  taken  from  Smith  (1980) 


o 


Sequence  Control 


Smith  suggests  that  menu  selection  might  be  used  as  an 
effective  means  of  providing  sequence  control  for 
uninitiated  users.  He  also  indicates  that  flexibility  is 
important  in  sequence  control,  particularly  in  interactions 
which  involve  the  modification  of  stored  data. 

o  User  Guidance 

Smith  suggests  that  the  fundamental  rule  in  the  area  of  user 
guidance  is  that  for  every  action  by  the  user  there  should 
be  a  response  by  the  machine.  Such  feedback  helps  maintain 
user  orientation. 


3. 3. 2. 4  Sidorsky  and  Parrish  Work  on  Battlefield  Automated 
Systems 

A  comprehensive  assessment  of  huma n-conputer  interaction  is 
currently  being  developed  by  Sidorsky  and  Parrish  (1980)  in 
work  conducted  for  the  Army  Research  Institute.  The  goal  of 
the  Sidorsky  and  Parrish  work  is  to  develop  general 
guidelines  for  describing  and  designing  battlefield 
automated  systems  so  that  ultimately  the  interoperat ibility 
of  such  systems  can  be  increased,  and,  consequently,  user 
errors  can  be  decreased.  Like  Smith  (1979)  and  Ramsey  and 
Atwood  (  19  79),  Sidorsky  and  Parrish  (1980)  have  developed  a 
conceptual  scheme  for  classifying  information  related  to 
human-ccmputer  interactions  (see  Table  3-16).  They  have 
also  developed  a  method  for  systematically  assessing  the 
human-ccmputer  transactions  which  currently  occur  in 
battlefield  systems. 


Table  3-16 


SIDORSKY’S  AND  PARRISH’S  SCHEME  FOR 
CLASSIFYING  HUMAN-COMPUTER  INTERACTION  INFORMATION' 


1.  CONTROL  METHODS 

1.1  Command  Languages 

1.2  Menus 

1.3  Function  Keys 

1.4  Hybrid  MethtxJs 

1.5  Prompt/HELP 

2.  DISPLAY  FORMAT 

2.1  Fixed  Alphanumeric  Displays 

2.2  Variable-Length  Alphanumeric 
Displays 

2.3  Graphic  Displays 

2.4  Highlighting 

3.  DATA  ENTRY  ASSISTANCE 

3.1  Information  on  Legal  Entries 

3.2  Unburdening  of  Input 

3.3  Interrupts  and  Work  Recovery 

4.  MESSAGE  COMPOSITION  AIDS 

4.1  System  Design  Features 

4.2  Format  for  .Alphanumeric 
Messages 

4.3  Graphic  Messages 


5.  DATA  RETRIEVAL  .ASSISTANCE 

5.1  Query  Method 

5.2  Query  Structure 

6.  GLOSSARIES 

6.1  Standard  Terms 

6.2  Character  Sets  and  Labels 

6.3  Glossary  Availability  and  Use 

6.4  Abbreviation  and  Coding 

7.  ERROR  HANDLING 

7.1  Prevention 

7.2  Detection 

7.3  Feedback 

7.4  Correction/Recovery 

8.  USER/OPERATOR  CONFIGURATIONS 

8.1  Operator(s)  Only 

8.2  Operator(s)  and  User(s) 

8.3  Combined  Operator/User 

8.4  Operator  and  User  Chains 


Table  derived  from  Sidorsky  and  Parrish  (1980). 


The  study  by  Sidorsky  and  Parrish  had  just  beyun  at  the  time 
of  the  ETES  review,  hence  they  only  had  developed  guidelines 
in  a  few  areas  (e.g.,  data  entry).  Thus,  their  work 
provided  relatively  little  guidance  for  the  development  of 
the  SDT. 


3.3.3  Summary  of  Implications  for  SDT 


As  a  result  of  the  review  of  literature  related  to  human- 
computer  interactions,  two  general  guidelines  were 
identified  and  subsequently  implemented  in  the  SDT. 


First,  in  selecting  the  type  of  dialogue  which  to  be 
implemented  in  the  SDT,  it  was  clear  that  some  form  of 
computer-initiated  dialogue  should  be  utilized  given  the 
types  of  uninitiated  users  who  could  be  expected  to  employ 
the  SDT.  As  a  result,  three  computer-initiated  dialogue 
types  —  question  and  answer,  form-filling  and  menu 
selection  were  selected  for  inclusion  in  the  SDT. 

Second,  coding  of  information  on  the  SDT  displays  was 
developed  in  accordance  with  the  most  current  guidelines  in 
Smith  (1979),  Sidorsky  and  Parrish  (1980),  and  Ramsey  and 
Atwood  (1980)  (see  Table  3-17).  However,  only  coding 
schemes  which  could  be  implemented  on  the  Apple  III  were 
utilized  (i.e.,  underlining,  italics,  character  size 
control,  position  displacement,  and  arrowing). 


Brightness  Control 
Character  Size  Control 
Ail  Upper  Case 
Reverse  Display 
Underlining 
Different  Font 
Color  Control 
Blinking,  Pulsating 
Boxing 
Arrowing 
Symbolic  Tagging 
Alphanumeric  Tagging 
Position  Displacement 


3.4  DEVELOP  SDT 

An  overview  of  the  process  used  to  develop  the  SDT  is 
presented  inn  Figure  3-2.  More  details  on  the  steps  in  this 
process  are  provided  in  the  sections  which  follow. 


3.4.1  Identify  SDT  Requirements 

Detailed  functional  requirements  were  developed  describing 
(1)  the  types  of  data  elements  which  should  be  included  in 
the  SDT,  (2)  the  types  of  huma n-canputer  dialogue  techniques 
to  be  used  in  the  SDT,  (3)  the  estimated  size  and  scope  of 
the  SDT,  and  (4)  the  type  of  communication  which  could  be 
expected  between  the  microcomputer  and  mainframe  in  the 
SDT.  Input  for  the  development  of  these  requirements  was 
provided  by  (1)  the  review  of  LSCMM  documents  and  processes, 
particularly  the  output  reports  generated  by  these  processes 
which  are  listed  in  Table  2-4  (see  Section  2.1),  (2)  the 

interviews  with  acquisition  participants  (see  Section  2.2), 
(3)  the  review  of  data  base  management  techniques  and 
capabilities  (see  Section  3.2),  and  (4)  the  review  of  human 
computer  interaction  guidelines  (see  Section  3.3). 

The  detailed  SDT  requirements  were  listed  in  the  first 
yearly  report  of  the  ETES  development  contract. 


3.4,2  Identify  Data  Elements 

A  detailed  data  dictionary  was  developed  describing  (1)  the 
title  of  each  data  element,  (2)  the  maximum  length  of  the 
data  element,  (3)  the  maximum  number  of  the  data  elements, 
(4)  the  type  of  data  contained  element  ( a  Ipha  nu.mer  i  c  or 


IDENTIFY 

SDT 

REQUIREMENTS 


Figure  3-2  Overview  of  Process  Used  to  Develop  SDT 


numeric,  (5)  the  category  of  the  data  element  (entity, 
subentity,  and  attribute)  (6)  the  relationships  between 
other  elements  in  the  data  structure  and,  (7)  the  definition 
to  be  used  to  describe  the  data  elements. 

The  data  dictionary  was  automated  to  facilitate  update.  The 
final  version  of  the  data  dictionary  is  listed  in  Appendix  A 
of  the  SDT  User's  Guide.  An  example  page  fran  the  data 
dictionary  is  presented  in  Table  3-18. 

3.4.3  Identify  Data  Structure 

Concurrent  with  the  identification  of  data  elements,  the  SDT 
data  structure  was  identified.  The  data  structure  specifies 
what  data  elements  are  entities,  subentities,  and  attributes 
(see  Section  1  for  a  description  of  these  concepts).  Seven 
major  entities  were  identified:  functions,  missions,  tasks, 
equipments,  duty  positions  (personnel),  media,  and  courses. 
Figure  3-3  describes  the  data  structure  that  was  used  to 
describe  the  relationships  between  these  entities.  A  more 
detailed  description  of  the  SDT  data  structure  is  provided 
in  Appendix  A  of  the  SDT  User's  Guide.  As  Figure  3-3, 
indicates,  the  SDT  employs  a  hierarchical  data  structure 
(see  Date  19  71  for  a  description  of  different  data  base 
structures).  Originally,  the  intent  was  to  use  a  n  n- 
hierarchical  or  relational  data  base  structure  ■'4iich  would 
allow  each  entity  to  be  associated  with  each  of  the  other 
entities.  However,  the  relational  data  base  structure 
required  internal  memory  req  ui  rarie  nts  that  were  beyond  that 
which  were  available  on  a  microcomputer. 
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Percent  performing  -  The  greater  the  proportion  of 
individuals  in  an  MOS  who  perform  a  task  on  the 
job,  the  greater  the  payoff  for  training  them  in 
an  institution.  Thus,  if  a  specific  duty  position 
accounts  for  a  substantial  percentage  of  the 
personnel  within  an  MOS,  then  that  duty  position 
is  identified  to  be  trained  in  the  institution. 
For  example,  for  MOS  95B,  Military  Police,  at 
Skill  Level  1,  there  are  several  different  duty 
positions,  such  as  military  police,  desk  clerk, 
radio  dispatch  clerk,  fingerprint  clerk,  machine- 
gunner,  investigator,  prisoner-of-war  processing 
specialist,  etc.  The  majority  of  these  duty 
positions  account  for  a  relatively  small  percent 
of  the  total  MOS  strength.  However,  the  training 
assignment  decision  is  easy  because  the  position 
of  military  police  accounts  for  approximately  60 
percent  of  the  MOS  for  Skill  Level  1. 

o  Time  between  training  and  required  job  performance 
-  If  a  task  is  complex  and  requires  practice  to 
maintain  proficiency  then  tasks  which  are  not 
performed  by  soldiers  within  6  months  after 
training  should  be  trained  in  the  unit. 

o  Consequences  of  Inadequate  Performance  -  This 
criteria  is  important  insofar  as  it  indicates 
which  tasks  may  need  concentrated  or  professional 
instruction  that  can  best  be  accomplished  in  an 
institutional  setting. 

Data  on  these  or  other  selected  criteria  can  either  be 
collected  in  surveys  conducted  directly  by  the  analyst  or 
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through  the  Army  Occupational  Survey  Program  (AOSP)  (more 
details  on  procedures  for  collecting  this  data  are  provided 
in  Section  3.1.)  These  data  can  be  summarized  on  a 
worksheet  like  that  shown  in  Table  3-10.  Data  entries  to 
this  worksheet  should  consist  of  mean  scores/ratings  for 
each  task  on  each  criteria. 

As  part  of  the  data  collection  process,  the  user  should  also 
examine  the  Trainer's  Guide  for  the  MOS.  This  guide  will 
define  the  specific  training  settings  currently  used  for  the 
MOS.  If  a  new  MOS  is  involved,  training  settings  for  other 
comparable  MOSs  should  be  used  (see  Section  2.0  for 
information  on  identifying  comparable  MOSs). 

3. 2. 2. 3  Make  Training  Setting  Assignments 

Training  setting  assignments  must  be  made  in  a  two  step 
process.  First,  tasks  should  be  assigned  to  the  either 
institutional  or  unit  training  setting  categories.  Second, 
specific  settings  within  each  of  these  general  categories 
must  be  determined.  It  should  be  noted  that  within  the 
institutional  training  setting  category,  there  is  a  close 
correspondence  between  specific  training  settings  and  skill 
level.  Thus,  if  you  know  the  skill  level  of  a  task  and  have 
made  the  assignment  to  the  institutional  training  category 
the  assignment  of  tasks  to  specific  institutional  training 
settings  is  largely  determined.  Listed  below  are  the 
relationships  between  skill  level  and  institutional  training 
settings. 


Skill  Level  1  -  o^„iC  Training,  Advanced 

Individual  Training  (AIT),  or  One  Station  Unit 
Training  (OSUT). 


o 


SDT  MODE  SELECTION 


SELECT  SDT  OPERATION  MODE: 


-EXIT  SDT- 
-AUDIT/UPDATE- 

-  INPUT  DATA- 
-CORRECT  DATA- 

-OUTPUT  DATA- 


SAVE/RESTORE- 

APPLICATIGNS- 

AUXILIARY- 


(Teririnate  the  Program) 

(Examine  Audit  Trail/Update  on  Honeywell 
at  DRC) 

(Add  Tabular  Data  into  the  SDT  data  files) 
(Change  Specific  Attribute  for  a  Particular 
Entity) 

(Display  or  Print  Existinq  Data) 


(Copy  this  system  to  or  from  dis)<ette) 
(Execute  the  Applications  Programs) 
(Execute  Auxiliary  SDT  Programs) 


Figure  3-4  Example  of  SDT  Menu 


Software  Documentation.  This  documentation  provides  a 
general  description  of  the  SDT  programs,  and  subroutines  , 
library  functions  and  program  logic. 


3.5  APPLY  SDT  TO  ARMY  WEAPON  SYSTEM 

The  original  objective  of  this  task  was  to  validate  the  SDT 
through  a  "simulated  system  development  exercise." 
Recognizing  the  limited  relevancy  associated  with  such  an 
exercise,  an  attempt  was  made  to  apply  the  SDT  to  a  "real" 
Array  weapon  systan  development  project.  With  this  in  mind, 
an  extended  set  of  discussions  was  held  with  the  Ballistics 
Missile  Defense  (BMD)  office  regarding  the  possible 
application  of  the  SDT  and  other  ETES  tools  to  the  manned 
BMD  subsystems.  Initial  talks  with  this  office  were 
extremely  positive  and  provisions  of  significant  additional 
funds  to  expand  the  ETES  application  seemed  likely.  However, 
the  BMD  office  finally  decided  that  an  ETES  application 
would  not  be  appropriate  since  (a)  the  BMD  system  concept 
had  been  changing  radically  and  had  not  yet  been  approved  by 
Congress,  and  (b)  existing  data  on  the  system  was  considered 
"sensitive"  and  not  appropriate  for  release  to  outside 
contractors . 

The  long  yet  unproductive  discussions  with  the  BMD  office 
when  coupled  with  the  tight  ETES  development  schedule,  make 
it  impossible  to  attempt  to  develop  another  extensive 
application  of  the  SDT.  Instead,  it  was  decided  to  examine 
the  SDT  in  a  limited  application  to  the  Single  Channel 
Ground  Activiated  Radar  System  (SINCGARS).  Since  a  HARDMAN 
methodology  was  being  conducted  on  SINCGARS  by  the  same 


contractor  who  was  developing  the  SDT  (DRC),  SINCGARS 
offered  a  low  cost  and  rapid  means  of  applying  the  SDT. 

During  the  SINCGARS  application,  the  SDT  was  used  to 
describe  system  functional  requirements,  tasks,  and 
equipments.  Three  criteria  were  used  to  assess  the  validity 
of  the  SDT  during  this  application:  ease  of  use,  adequacy  of 
SDT  as  a  Data  Base  Management  tool,  and  meaningfulness  of 
results.  Ease  of  use  was  assessed  by  interviewing  the 
personnel  who  applied  the  SDT.  Mean ing ful ness  of  results 
and  data  base  adequacy  was  assessed  by  examining  the  data 
input  into  the  SDT.  The  results  of  the  interviews  indicated 
that  the  SDT  was  easy  to  use  and  that  it  could  adequately 
perform  the  data  base  functions  of  storing,  updating,  and 
outputting  system  data.  The  major  user  of  the  SDT  was  a 
former  Army  training  analyst  who  was  able  to  obtain 
meaningful  output  for  an  SDT  entity  within  one  week  of  his 
initial  session  on  the  computer;  -  this  despite  the  fact 
that  he  was  provided  with  no  user  guide;  (the  user's  guide 
was  developed  in  a  later  task),  and  with  only  a  twenty 
minute  briefing  on  SDT  operation. 

However,  mea n i ng ful ness  of  the  results  produced  during  the 
application  appeared  to  be  only  fair.  More  specifically, 
there  appeared  to  be  some  confusion  regarding  the  meaning 
and  use  of  several  of  the  non-training  related  SDT  data 
elements,  such  as  system  functions  and  missions.  This 
problem  seems  reasonable  since  the  ETES  user  guides  did  not 
exist  and  there  were  no  other  documents  describing  the  use 
of  several  of  these  non-training  related  data  elements. 


In  summary,  the  results  of  the  SDT  application  clearly 
indicated  that  (1)  the  SDT  was  indeed  easy  to  use  and  (2) 


successful  usage  of  the  SDT  and  all  its  associated  data 
elements  would  require  users  to  either  be  familiar  with  ETES 
procedures  which  generate  the  SDT  data  elements  or  to  have 
the  user's  guides  for  these  procedures  on-hand  for  reference 
throughout  the  application  of  the  SDT. 


SECTION  4  -  TASK  3:  DEVELOP  TRAINING  ESTIMATION  AIDS 

AND  PROCEDURES 


Figure  4-1  displays  the  procedures  that  were  used  to  develop 
the  Training  Estimation  Aids  and  Procedures.  More  details 
on  each  of  these  steps  are  described  in  the  sections  which 
follow . 


4.1  IDENTIFY  FUNCTIONAL  REQUIREMENTS 

Based  upon  the  analysis  of  LCSMM  requirements  and  user  needs 
which  was  conducted  in  Task  1,  a  detailed  listing  of  the 
functions  which  a  comprehensive  set  of  early  training 
estimation  aids  and  procedures  must  achieve  was  identified. 
This  list  is  presented  in  Table  4-1.  The  original  intent  of 
Task  4,  as  outlined  in  the  statement  of  work,  was  to  simply 
develop  aids  and  procedures  for  generating  task 
descriptions.  However,  the  results  of  the  Task  1  review  of 
existing  Army  procedures  indicated  that  an  early  training 
estimation  must  perform  six  basic  functions.  These 

functions  are: 

(1)  Estimate  and  describe  a  complete  list  of  system 
functions.  Without  a  complete  list  of  system 
functions  and  associated  performance  goals  it  is 
extremely  difficult  to  build  a  comprehensive 
system  description  or  to  evaluate  how  well  the 
description  meets  requirements.  Functional 


TASK  1 
RESULTS 


functions  Not  Selected  for  Inclusion  in  Training  Estimation  Aids/Procedures 


information  items  are  identified  as  part  of  the  QPOI 
construction  process  (Procedure  3.6),  identification  of 
course  locations  is  relatively  straightforward.  Additional 
guidance  for  course  location  identification  can  be  provided 
by  examining  (a)  the  course  locations  used  in  comparable 
existing  courses  and  (b)  the  projected  locations  where  the 
weapon  system  will  be  fielded. 

4.1.2  Determine  Course  Frequency 

Course  frequency  (i.e.,  the  number  of  starts  and  Sessions 
per  site)  is  determined  by  applying  the  algorithm  listed  in 
Table  4-1.  To  estimate  course  frequency,  you  must  determine 
the  number  of  students  to  be  trained  (Procedure  4.2).  The 
algorithm  listed  in  Table  4-1  must  be  applied  for  each 
location  and  year  in  which  the  course  will  be  given. 

4.1.3  Determine  Usage  Rates 

Usage  rates  are  determined  for  instructors  and  training 
media.  The  usage  rate  for  instructors  is  called  tJi^.- 
student/instructor  ratio.  The  usage  rate  for  media  is 
called  the  student/media  ratio. 

Student/instructor  ratios  are  determined  for  each 
instructional  method  used  in  a  course.  Table  4-2  displays 
the  student/instructor  ratios  that  are  provided  for  each 
type  of  instructional  method  in  DA  Pam  570-558  and  TRADO? 
Cir  351-12. 

S  tude  n  t /''med  i  a  rates  are  determined  by  identifying  til'.'  raid’s 
that  were  used  for  similar  types  of  media  in  com-ar  ‘'rp, 
courses.  This  information  may  be  obtained  by  examinii,  |  thi.' 
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TABLE  4-1. 


ALGORITHM  FOR  DETERMINING  NUMBER  OF  COURSE 
STARTS  AND  SESSIONS  PER  SITE. 


1.  Establish  number  of  courses  required  per  site 


.  STUDIN 
NSITE 


NMA.'C  =  NCCURSE 


STUDIN 

NSITE 

NI4AX 

NCOURSE 


=  Student  Input  Requirements 
=  Number  of  Training  Sites 
=  Maximum  Student-Instructor  Ratio 
for  Course 

=  Number  of  Courses  Required  Per 
Site 


2. 


Determine  maximum  number  of  starts  per  year 
(CIAX) 


50 

CLENGTH 


CMAX 


CLENGTH  =  Course  Length  in  weeks 

CMAX  =  Maximum  Number  of  Course  Starts 

per  year  (NCON) 


3.  Select  the  smaller  of  NCOURSE  or  CMAX  as  the 
number  of  course  starts  (NCON)  required  per 
site . 


4.  Determine  maximum  number  of  sessions  needed  per 
start  (NSES) 

If  NCOURSE  CMAX,  NSES  =  1 

If  NCOURSE  CMAX,  ^  NSES 

Round  NSES  up  to  next  highest  unit. 
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between  the  Army  HARDMAN  applications  and  ETES.  EXES 
procedures  for  determining  system  functional  requirements, 
estimating  training  programs  and  estimating  training 
resources  and  costs  were  incorporated  into  HARDMAN.  In  many 
cases,  since  HARDMAN  was  being  continuously  applied  to  Army 
systems  during  ETES  development  ETES  developed  procedures 
were  applied  in  Army  HARDMAN  applications  before  these 
procedures  were  fully  completed  and  integrated  in  ETES. 

TEEM  provided  two  critical  inputs  to  ETES.  First,  it 
provided  a  media  selection  model,  which  with  significant 
modifications,  provided  an  effective  vehicle  for  early  media 
selection  (see  the  description  of  the  ETES  Media  Selection 
Program  in  Appendix  A).  Second,  the  TEEM  concept  of 
training  efficiency  provided  a  useful,  albeit  crude  measure 
for  estimating  the  "effectiveness"  or  goodness  of  an  early 
concept  for  a  training  program  element.  More  specifically, 
in  the  training  efficiency  concept,  the  goodness  of  a 
training  program  element  (such  as  media)  is  measured  by 
assessing  how  well  the  characteristics  of  the  tasks  to  be 
trained  match  the  characteristics  of  the  training  program 
e  lement. 

After  existing  tools  were  selected  for  inclusion  in  the 
training  estimation  aids  and  procedures,  the  functions  were 
examined  to  identify  the  areas  where  state  of  the  art 
technology  could  be  most  effectively  applied  to  develop  new 
tools.  In  determining  whether  a  new  tool  was  appropriate, 
three  factors  were  considered:  (a)  the  criticality  of  the 
function  to  early  training  estimation,  (b)  the  technical 
feasibility  of  developing  the  new  tool/given  the  current 
state  of  the  art,  and  (c)  the  resources  available  under  the 


ETES  contract.  Table  4-1  lists  the  functional  areas  where 
new  tools  were  developed. 

There  were  five  major  functional  areas  where  state-of-the- 
art  technology  and  resource  constraints  prohibited  the 
development  of  a  new  training  estimation  aids  and/or 
procedure.  First,  it  was  not  possible  to  develop  techniques 
for  estimating  collective  training.  Second,  it  was  not 
possible  to  develop  procedures  for  estimating  unit 
training.  Thus,  the  ETES  Training  Estimation  Aids  and 
Procedures  developed  in  this  contract  focused  on  individual 
institutional  training  (see  Table  4-2).  It  was  decided  that 
it  was  more  beneficial  to  develop  a  complete  set  of 
procedures  for  dealing  with  individual  institutional 
training  which  covered  the  entire  training  estimation 
process  from  task  enumeration  to  training  program  evaluation 
than  to  develop  a  partial  set  of  procedures  for  the  full 
range  of  Army  training  settings.  Third,  procedures  were  not 
developed  for  estimating  acquisition  costs  for  training 
programs  (thus,  only  procedures  for  estimating  operations 
and  support  costs  were  developed).  The  major  reason  for 
this  was  that  it  was  extremely  difficult  to  obtain  any  kind 
of  systematic  data  on  training  acquisition  costs.  More 
specifically,  it  was  very  difficult  to  obtain  any  kind  of 
useful  data  for  existing  training  programs.  It  was 

determined  that  while  some  data  on  acquisition  costs  is 
available  in  informal  data  bases  throughout  the  Army,  the 
cost  of  locating  and  analyzing  this  data  would  have  expended 
too  much  of  available  ETES  development  resources. 

Fourth,  procedures  were  not  developed  for  providing  input 
lata  to,  or  evaluating  the  results  from.  Operational  Testing 
(OT)  even  though  these  functions  were  clearly  related  to 


Table  4-2  Focus  of  Current  ETES  Training  Estimation 
Aids/Procedures 


training  estimation  during  the  Demonstration  and  Validation 
phase  of  the  acquisition  process.  The  reason  for  this  was 
that  a  separate  project,  the  Human  Resource  and  Test 
Evaluation  System  (HRTES),  was  designed  to  provide 
procedures  for  providing  manpower,  personnel  and  training 
related  input  into  Operational  Testing  (OT) .  However, 
guidelines  were  developed  for  providing  input  to  HRTES. 
HRTES  and  ETES  and  other  manpower,  personnel  and  training 
tools  are  being  integrated  in  the  MIST  contract.  (see 
Appendix  D  for  a  description  of  HRTES  and  MIST). 

Fifth,  while  procedures  were  developed  for  estimating 
training  efficiency/effectiveness,  development  of  a  complete 
capability  for  estimating  early  effectiveness  was  not 
possible.  More  specifically,  it  was  not  possible  to  develop 
measures  of  effectiveness  that  were  appropriate  for 
inclusion  in  cost/benefit  ratios.  Such  ratios  require  the 
development  of  effectiveness  criteria  which  can  be  measured 
on  a  true  ratio  scale.  It  was  not  possible  to  develop 
procedures  which  could  produce  effectiveness  measures  which 
met  all  the  theoretical  requirements  for  ratio  scale 
development . 


4.3  SELECT  TOOLS  TO  BE  AUTOMATED 


Once  potential  tools  were  identified,  an  analysis  was 
conducted  to  identify  which  tools  should  be  automated.  Each 
tool  which  was  amenable  to  automation  was  identified  and 
assigned  a  priority,  the  intent  being  to  automate  tools 
based  on  their  priority  until  funds  allotted  for  this  task 
were  expended  (No  specific  set  of  procedures  were  mentioned 
in  the  statement  of  work).  Priority  was  based  on  three 


factors  (a)  how  critical  the  tool  was  to  early  training 
estimation,  (b)  the  amount  of  training  analyst  time  that  I 

would  be  expected  to  be  saved  by  automating  the  tools  and 
(c)  ease  with  which  the  tool  could  be  automated,  given  the 
hardware/  software  configuration  selected  for  the  SDT  (Apple  i 

III/PASCAL).  The  overall  goal  in  operating  the  tools  was  to  I 

get  the  "biggest  bang  for  the  buck"  while  recognizing  that 
additional  automation  would  take  place  in  the  ETES 

implementation  contract. 

Two  automated  tools  were  developed;  one  for  assigning  tasks 
to  media  based  on  efficiency  of  the  training  media  and  one 
for  estimating  operating  and  support  costs  for  institutional 
training  courses.  An  overview  of  these  two  tools  is 
provided  in  Appendices  A  and  B.  A  more  detailed  description 
of  these  two  automated  tools  is  provided  in  the  User's  Guide 
Media  Selection  Program  and  User's  Guide;  Automated  Resource 
and  Cost  Estimation  Technique. 


4.4  DEVELOP  TRAINING  ESTIMATION  AIDS  AND  PROCEDURES 

Manual  and  automated  training  estimation  aids  and  procedures 
for  the  functional  areas  selected  in  the  previous  subtasks 
were  developed.  The  two  automated  aids  were  developed  using 
the  procedures  outlined  in  Figure  4-2.  A  description  of  the 
Training  Estimation  Aids  and  Procedures  which  were  developed 
under  this  subtask  is  provided  in  Section  1.5.2. 
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Figure  4-2  Procedures  for  Developing  Automated 
Training  Estimation  Aids/Procedures 
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SECTION  5  -  TASK  4;  DEVELOP  EVALUATIVE  TECHNOLOGY 


Two  major  activities  were  accomplished  under  this  task. 
First,  a  review  was  conducted  of  existing  simulation 
techniques  to  detei  une  if  any  of  these  techniques  could  be 
used  in  early  training  estimation.  Second,  the  Evaluative 
Technology  component  of  ETES  was  constructed.  More  details 
on  these  two  activities  are  presented  in  the  sections  which 
follow . 


5.1  REVIEW/EXAMINE  SIMULATION  TECHNIQUES 

A  detailed  review  was  made  of  existing  simulation  techniques 
to  determine  if  any  of  these  techniques  had  applicability 
for  early  training  estimation. 

The  purpose  of  this  review  was  to  examine  "low  cost  methods 
for  simulating  interacting  tasks  together  with  combat 
scenarios."  Ideally,  the  simulation  technique  would  receive 
as  input  system  descriptions  from  the  SDT  and  produce  as 
outputs  a  listing  of  the  criticality  of  each  human  task  to 
overall  system  performance.  Thus,  the  simulation  technique 
would  have  the  capability  of  relating  both  system  design 
concepts  and  training  program  concepts  to  task  performance 
and  relating  task  performance  to  overall  system  performance. 

Table  5-1  lists  the  simulation  techniques  which  were 
reviewed  during  this  task.  The  two  most  premising 

techniques  identified  during  this  review  were  the  Systems 
Analysis  of  Integrated  Networks  of  Tasks  (SAINT)  developed 


by  Pritsker  and  Human  Operator  Simulation  (HOS)  Technique 
developed  by  Streob,  Glenn  and  Whery  (1978).  However,  an 
examination  of  these  two  techniques  indicated  that  they  both 
required  detailed  task  data  as  input.  This  type  of  detailed 
task  data  is  generally  not  available  during  the  early  phases 
of  the  acquisition  process.  Consideration  was  given  to 
modifying  one  of  these  techniques,  SAINT,  to  accept  more 
generalized  task  input.  However,  this  approach  was  not 
taken  for  two  reasons.  First,  advanced  development  work  on 
a  human  operator  simulation  using  SAINT  was  being  conducted 
in  ARI's  MOPADs  contract.  It  was  felt  that  this  work  would 
make  previous  SAINT  work  obsolete  and  that  it  made  more 
sense  to  wait  until  this  more  advanced  MOPADs  simulation 
model  was  complete  and  then  modify  this  model  for  early 
simulation  purposes  than  to  modify  a  more  primitive  SAINT 
model.  Second,  and  most  importantly,  the  review  of  existing 
Army  early  training  estimation  needs  conducted  during  Task  1 
indicated  that  there  were  more  real  and  pressing  needs  for 
evaluating  early  training  programs  with  respect  to  other  key 
criteria  such  as  resource  requirements,  cost,  and  training 
efficiency/effectiveness  than  there  were  to  evaluate  these 
programs  with  respect  to  task  criticality  (several  less 
soph  is  t  ica  ted  procedures  for  estimating  task  criticality 
were  readily  available  and  were  eventually  incorporated  into 
the  ETES  Training  Estimation  Aids  and  Procedures). 


5.2  DEVELOP  EVALUATIVE  TECHNOLOGY 


An  overview  of  the  procedures  that  were  useij  to  develop  the 
Evaluative  Technology  is  provided  in  Figure  5-1.  A  more 
letailed  description  of  these  procedures  is  provided  in  the 
section  which  follows. 


Figure  5-1  Procedures  for  Developing  Evaluative  Technology 
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5.2.1  Identify  Functional  Requirements 

Based  upon  the  analysis  of  LCSMM  requirements  and  user  needs 
identified  in  Task  1,  it  was  determined  that  an  effective 
Evaluative  Technology  for  evaluating  early  training  programs 
must  contain  procedures  for  (1)  developing  f  igures-of-me  r it 
for  assessing  the  integrated  impacts  of  the  training 
requirements  developed  in  the  Training  Estimation 
Aids/Procedures,  (2)  identifying  potential  problem  areas  for 
system  training  and  the  likely  sources  of  these  problems, 
(3)  identifying/evaluating  training  alternatives,  which  can 
be  expected  to  reduce  the  training  problems,  (4)  developing 
tra ini ng-related  input  to  key  acquisition  documents,  and  (5) 
determ  i  ning/evalua  t  ing  training  development  schedules  (see 
Figure  5-2) 

5.2.2  Identify  Functions  and  Tools  to  be  Included  in 
Evaluative  Technology 

Once  the  functions  required  for  evaluating  early  training 
programs  were  identified,  existing  tools  and  technologies 
were  examined  to  (1)  identify  any  existing  tools  which  could 
be  modified  to  meet  the  early  training  evaluation  functions, 
and  (2)  identify  the  most  premising  areas  for  new 
development . 

In  order  to  be  selected  for  inclusion  in  ETES,  a  tool  had  to 
meet  two  general  requirements.  First,  it  had  to  be  capable 
of  producing  or  being  modified  to  produce  output  that  was 
appropriate  for  the  early  training  of  the  acquisition 
process.  Second,  the  tool  had  to  be  capable  of  being 
applied  with  the  types  of  general  data  and  information  that 


are  available  during  the  early  phases  of  the  acquisition 
process . 


Table  5-2  lists  the  tools  which  met  the  criteria  for 
inclusion  in  the  Evaluative  Technology. 

After  existing  tools  were  investigated,  the  functions  were 
examined  to  identify  the  areas  where  state-of-the-art 
technology  could  be  effectively  applied  to  develop  the  new 
tools.  In  determining  whether  a  new  tool  was  feasible, 
three  factors  were  considered:  (a)  the  criticality  of  the 

function  to  early  training  estimation,  (b)  the  technical 
feasibility  of  developing  the  new  tool  given  the  current 
state-of-the-art,  and  (c)  the  resources  available  under  the 
ETES  contract.  Table  5-2  lists  the  functional  areas  where 
new  tools  were  developed. 

As  Table  5-2  indicates,  it  was  possible  to  develop  a  tool 
for  each  of  the  functional  areas  related  to  training 
evaluation.  However,  in  two  of  the  functional  areas  it  was 
not  possible  to  develop  tools  which  fully  met  all 
requirements  associated  with  these  functions. 

First,  in  the  functional  area  involving  the  development  of 
f  ig ur es-of -me r i t ,  procedures  were  identified  for  developing 
estimates  for  eight  f  igures-of-mer i t  (training  cost, 
training  efficiency,  training  effectiveness,  congruence  with 
training  development  guidelines,  congruence  with  program 
requirements,  training  complexity,  training  capacity  and 
feasibility  of  the  training  program).  However,  only  a  very 
simplified  procedure  was  identified  for  aggregating  these 
scores.  There  are  several  more  soph i st i ca ted  techniques 
'  such  as  mu  1 1  1  a  1 1  r  i  bu  te  utility  analysis  which  coulil  be  used 


Table  3-2  FUNCTIONAL  REQUIREMENTS  FOR  EVALUATIVE  TECHNOLOGY 


Functions  Selected  for  Inclusion  in  Evaluative  Technology 
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to  aggregate  these  scores  but  these  techniques  require  more 
sophisticated  subjective  estimation  methods.  Modification 
of  such  methods  for  use  during  the  early  phases  of  the 

acquisition  process  was  not  feasible  within  the  resource 
constraints  of  the  ETSS  development,  contract. 

Also  within  the  same  functional  area  of  developing  training 
f igures-of-merit,  it  was  not  possible  to  develop  a  figure-of 
merit  which  would  provide  a  measure  of  the  expected  impact 
of  the  training  concept  on  system  performance.  During  the 
early  phases  of  the  acquisition  process,  the  only  way  to 

develop  such  a  measure  is  to  use  a  simulation  model.  As 
section  5.1  indicated,  development  of  such  a  model  was  not 
possible  within  the  resource  constraints  of  the  ETES 

development  contract. 

Finally,  it  was  only  possible  to  develop  crude  procedures 
for  identifying  the  likely  causes  of  training  problems. 
Development  of  more  sophisticated  procedures  required  the 
development  of  a  quantitative  model  (preferably  a  simulation 
model)  for  relating  training  program  elements  to  the 

training  figures  of  merit.  Again,  such  a  complicated  model 
was  beyond  the  scope  of  the  present  ETES  development  effort. 


5.2.3  Select  Tools  To  Be  Automated 

Once  potential  tools  were  identified,  an  analysis  was 
conducted  to  identify  which  tools  should  be  automated.  Each 
tool  which  was  amenable  to  automation  was  identified  and 
assigned  a  priority,  the  intent  being  to  automate  tools 
based  on  ;heir  priority  until  funds  available  for  this  task 
were  exper:ded.  Priority  was  based  on  three  functions:  (a) 
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how  critical  the  tool  was  to  early  training  estimation,  (b) 
the  amount  of  time  that  would  be  expected  to  be  saved  by 
automating  the  tool,  and  (c)  the  ease  with  which  the  tool 
could  be  automated  given  the  hardware/software  configuration 
selected  for  the  SDT  (Apple  III/PASCAL). 

Within  the  resource  constraints  of  the  ETES  contract,  it  was 
only  possible  to  select  one  additional  tool  for  automation, 
the  Automated  Training  and  Planning  and  Scheduling  Technique 
(APST).  An  overview  of  this  program  is  provided  in  Appendix 
C.  However,  procedures  were  also  identified  for  using  the 
automated  Resource  and  Cost  Estimation  Technique  developed 
as  part  of  the  ETES  Training  Estimation  Aids/Procedures  to 
conduct  sensitivity  analysis. 

5.2.4  Develop  Evaluative  Technology 

Potential  automated  aids  and  procedures  which  were 
identified  in  the  previous  step  were  developed.  A 
description  of  these  aids  and  procedures  is  provided  in 
Section  1.5.3. 


SECTION  6  -  TASK  5:  DEVELOP  USER'S  GUIDE 


During  this  task,  user's  guides  were  developed  for  the  three 
ETES  components  (SDT,  Training  Estimation  Aids  and 
Procedures,  and  Evaluative  Technology)  and  a  final  report 
was  constructed  to  describe  the  research  conducted  during 
the  ETES  study. 

Table  6-1  summarizes  the  documents  developed  during  this 
task. 


To  facilitate  independent  application  of  the  four  automated 
ETES  tools  (SDT,  Media  Selection  Program,  Automated  Resource 
and  Cost  Estimation  Technique,  and  Automated  Training 
Planning  and  Scheduling  Technique),  separate  users'  guides 
were  developed  for  each  of  these  tools.  In  addition  the 
ETES  manual  procedures  were  consolidated  in  a  single  user's 
guide  (see  Table  6-1).  Each  of  users's  guides  was  designed 
to  provide  detailed  step-by-step  procedures  for  using  the 
ETES  tools. 


Table  6-1  ETES  Documentation 


•  Final  Report  (150  pages) 

•  ETES  Procedural  Guide  (300  pages) 

•  User's  Guide:  System  Description  Technology  (300  pages) 

•  User's  Guide:  Media  Selection  Program  (200  pages) 

•  User's  Guide:  Resource  and  Cost  Estimation  Technique  (100  pages) 

•  User's  Guide:  Automated  Training  Planning  and  Scheduling  Technique 

(100  pages) 


•  ETES  Software  Documentation  (300  pages) 


SECTION  7-  POTENTIAL  IMPROVEMENTS  TO  CURRENT 

ETES  CONCEPT 


The  ETES  development  study  was  designed  to  provide  an 

initial  capability  for  developing  early  estimates  of 

training  requirements.  As  Sections  3  thru  6  have  indicated, 
resource  constraints  prohibited  the  full  development  of 

tools  in  several  key  areas. 

This  section  briefly  describes  the  areas  in  which  the 
current  ETES  can  be  improved  to  (1)  provide  more  accurate 
estimates  of  training  requirements  or  (2)  provide  estimates 
of  training  requirements  in  a  more  timely  and  cost-effective 
f  ash  ion. 

Table  7-1  summarizes  the  major  improvements  which  may  be 

made  to  each  component  of  ETES.  A  more  detailed  description 
of  these  improvements  is  provided  in  the  sections  which 
follow. 


7,1  IMPROVEMENTS  TO  SYSTEM  DESCRIPTION  TECHNOLOGY 

There  are  four  general  areas  in  which  the  System  Description 
Technology  could  be  improved.  First,  the  mainframe 
functions  of  the  SDT  should  be  expanded  and  refined  to 
provide  a  full  capability  for  distributed  processing  of 
data.  These  mainframe  functions  include  techniques  for  (1) 
storing  a  centralized  copy  of  the  SDT,  (2)  transmitting  and 
communicating  data  to  various  users,  (3)  providing  security 
for  data  base  access,  and  (4)  keeping  an  audit  trail  of 


system  changes.  The  current  version  of  ETES  only  provides 
relatively  simple  capabilities  in  each  of  these  areas.  More 
sophisticated  capabilities  were  not  developed  because  it  was 
not  possible  to  identify  the  "mainframe"  computer  on  which 
the  SDT  would  be  implemented.  Since  the  development  of  the 
SDT,  however,  the  Army  Research  Institute  (ARI)  has  selected 
the  VAX  11/780  as  the  mainframe  computer  on  which  the  SDT 
and  other  components  of  the  ETES  will  be  implemented.  With 
the  identification  of  the  mainframe,  more  sophisticated 
software  can  be  developed  for  the  SDT  mainframe  functions. 

Second,  the  current  ETES  only  has  limited  capabilities  for 
searching  and  sorting  data.  To  better  meet  the  unique 
demands  of  individual  users,  a  more  flexible  searching  and 
sorting  capability  should  be  developed. 

Third,  more  comprehensive  graphics  capabilities  must  be 
added  to  the  SDT.  The  current  SDT  can  produce  reports  that 
contain  the  the  data  elements  needed  in  many  key  acquisition 
documents.  However,  it  can  only  output  that  data  in  formats 
that  "look  like"  the  acquisition  document. 

Fourth,  the  SDT  data  elements  should  be  expanded  to  meet  the 
needs  of  any  additional  tools  which  are  developed. 
Currently,  for  example,  as  part  of  the  MIST  project,  the 
prototype  SDT  is  being  expanded  to  handle  the  data  required 
to  support  manpower  and  personnel  estimation  tools. 


7.2  IMPROVEMENTS  TO  TRAINING  ESTIMATION  AIDS/PROCEDURES 

There  are  four  general  areas  in  which  the  ETES  Training 
Estimation  Aids/Procedures  can  be  improved.  First, 


procedures  must  be  developed  for  estimating  collective 
tasks,  collective  training  programs,  collective  training 
resources,  collective  training  costs,  and  collective 
training  effectiveness.  Second,  procedures  must  be 

developed  for  estimating  training  programs,  resources  and 
costs  for  individual  unit  training.  These  two  improvements 
will  became  increasingly  important  given  the  current  trend 
toward  shifting  more  and  more  training  to  the  unit.  Third, 
procedures  must  be  developed  for  developing  more  detailed 
and  more  accurate  measures  of  training  effectiveness.  The 
current  ETES  procedures  only  provide  crude  estimates  of 
training  effectiveness.  While  appropriate  for  the  very 
earliest  phases  of  the  acquisition  process,  these  procedures 
do  not  provide  the  type  of  detailed  data  which  is  needed  for 
the  refined  training  tradeoff  analyses  which  occur  in  the 
later  phases  of  the  acquisition  process. 

Fourth,  resource  constraints  prohibited  the  develqpment  of 
automated  aids  for  several  key  functions.  Automated  aids 
can  assist  the  training  analyst  in  dealing  with  the  frequent 
design  changes  which  occur  during  the  earliest  phases  of  the 
acquisition  process.  Some  of  the  current  TEAP  functions 
which  are  most  amenable  to  automation  are: 

3.  1  Select  Tools  for  Training 

3.4  Assign  Tasks  to  Training  Settings 

4.2  Determine  Number  of  Students  to  Be  Trained 

4.4  Determine  Facilities  Requirements 

4.6  Determine  Other  Training  Resources 

5.2  Aggregate  Course  Costs 


7.3  IMPROVEMENTS  TO  EVALUATIVE  TECHNOLOGY 

These  ace  four  general  areas  where  the  Evaluative  Technology 
may  be  improved.  First,  the  current  Evaluative  Technology 
only  employs  some  rather  simplified  techniques  for 
aggregating  individual  training  figures  of  merit.  While 
simple  to  use  and  appropriate  for  the  very  earliest  phases 
of  the  acquisition  process,  these  techniques  do  not  have  the 
accuracy  needed  for  the  detailed  tradeoff  and  sensitivity 
analyses  which  occur  in  later  phases  of  the  acquisition 
process.  Second,  a  more  sophisticated  tool  must  be 

developed  for  identifying  the  causes  of  training  problems. 
One  possible  tool  for  performing  this  complex  function  is  a 
conputer  simulation  model.  As  noted  in  Section  6, 
development  of  such  a  model  will  not  be  easy  since  most 
simulation  models  have  extensive  input  data  requirements.  An 
alternative  approach  is  to  use  an  artificial  intelligence 
technique  to  build  an  "expert  system"  for  diagnosing 
training  problems  similar  to  the  MYCIN  system  which  was 
developed  to  diagnose  medical  problems  (see  Winston,  19  77 
for  an  overview  of  MYCIN). 

Third,  an  automated  worksheet  was  developed  for  estimating 
selected  training  resources  and  course  costs.  This 
worksheet  makes  it  very  easy  to  change  data  and  immediately 
assess  impacts.  The  automated  worksheet  concept  must  be 
expanded  to  cover  other  ETES  functions  and  individual 
automated  aids  must  be  linked  together  to  provide  a  more 
systematic  capability  for  assessing  impacts. 

Fourth,  procedures  must  be  developed  for  interfacing  ETES 
with  techniques  which  are  used  in  the  later  phases  of  the 
acquisition  process  such  as  instructional  system  development 


7.4  GENERAL  AREAS  OF  IMPROVEMENTS 


There  are  three  general  areas  which,  although  outside  the 
scope  of  the  current  ETES  concept,  could  significantly 
improve  the  use  of  ETES  products.  First,  to  facilitate  the 
transmission  and  communication  of  data,  the  SDT  data  base 
management  system  must  be  incorporated  into  a  Management 
Information  System  for  Army  manpower,  personnel  and  training 
analysts  (this  will  in  fact  be  done  in  later  phases  of  the 
MIST  project  )  . 

Second,  a  comprehensive  tool  must  be  developed  for  assessing 
the  integrated  impacts  of  manpower,  personnel,  and  training 
requirements  and  for  conducting  integrated  tradeoffs  among 
these  three  closely  linked  system  elements. 

Third,  a  systematic  tool  must  be  developed  for  estimating 
the  impact  of  the  alternative  manpower,  personnel,  and 
training  concepts  on  overall  system  perfonnance.  To  be 
useful,  this  tool  must  not  have  extensive  input  data 
requirements.  Otherwise,  it  will  never  be  used  during  the 
earliest  phases  of  the  acquisition  process,  when  most  major 
MPT  and  system  design  decisions  are  made. 


APPENDIX  A:  OVERVIEW  OF  MEDIA  SELECTION  PROGRAM 


This  appendix  provides  a  brief  overview  of  the  Media 
Selection  Program.  A  more  detailed  description  is  provided 
in  the  User's  Guide:  Media  Selection  Program.  The  Media 
Selection  Program  is  an  automated  tool  for  (1)  assigning 
tasks  to  media  and  (2)  calculating  the  efficiency  and 
effectiveness  of  various  task-media  combinations.  The  Media 
Selection  Program  allows  users  to  assign  tasks  to  media  in  a 
manner  that  maximizes  overall  efficiency,  maximizes  overall 
"effectiveness,"  or  minimizes  overall  cost.l  In  addition. 
It  allows  users  to  assign  tasks  to  media  in  a  manner  that 
optimizes  various  combinations  of  these  variables,  including 
an  overall  "utility"  measure  which  combines  either  cost  and 
efficiency  or  cost  and  effectiveness. 


More 


Efficiency,  in  the  Media  Selection  Program,  is  determined  by 
comparing  the  stimulus,  response,  and  feedback 
characteristic  of  the  tasks  to  the  stimulus,  response,  and 
feedback  characteristics  of  potential  media.  More 
specifically,  a  score  is  calculated  which  describes  the 
match  between  media  and  tasks  on  these  characteristics. 
Efficiency  for  each  task-media  combination  is  calculated  by 
dividing  this  score  by  the  maximum  match  that  may  be 


The  program  uses  a  relative  cost  and  not  actual  cost  to 
measure  the  potential  cost  requirements  of  media.  In 
aduininn,  the  progrim  does  not  use  a  measure  of 
"effectiveness"  thar  fits  the  most  common  usage  of  that 
t-rms.  Ritner  effectiveness  is  actually  efficiency  weignted 


achieved  for  the  task.  Total  efficiency  for  a  set  of  tasks 
is  the  sum  of  the  efficiency  score  for  individual  tasks. ^ 

"Effectiveness"  is  calculated  by  weighting  the  efficiency  of 
each  task  by  a  task  criticality  score.  The  task  criticality 
score  is  a  user-defined  weighted  combination  of  the  eight 
task  factors  typically  used  in  selecting  tasks  for  training. 
These  eight  factors  are  task  frequency,  percent  members 
performing,  percent  time  performing,  task  delay  tolerance, 
consequences  of  inadequate  performance,  task  learning 
difficulty,  probability  of  deficient  performance,  and  time 
between  entry  and  performance. 

A  matrix  of  relative  cost  values  is  stored  in  the  program 
for  each  major  media  category.  In  addition,  a  built-in  set 
of  algorithms  is  used  to  produce  the  utility  measure  which 
combines  cost  with  either  effectiveness  or  efficiency. 

PROCEDURAL  OVERVIEW 

The  Media  Selection  Program  is  an  interactive  menu-driven 
system.  This  means  that  users  do  not  have  to  know  or  use  a 
computer  language  to  run  the  program.  Instead,  they  can  run 
through  the  program  by  selecting  options  from  a  series  of 
menu  s . 


This  concept  of  meii  i  a  o(-i),'iency  was  originally 
developed  by  Jorgensen  i  and  was  incorporated  into 

Training  Efficiency  Estimation  Mooe!  (TEEM)  described  in 
J  o  r  g  e  n  s  e  n  ,  K  u  b  u  1  a  ,  ,a  n  d.  t  1  a  s  ■:  I  S  1  )  . 


An  overview  of  the  procedures  for  using  the  Media  Selection 
Program  is  provided  in  Figure  A-1.  To  begin  the  procedures, 
users  must  rate  each  task  on  (1)  the  psychological  variables 
to  be  used  to  assess  the  match  between  tasks  and  media,  and 
(2)  rhe  variables  to  be  used  to  assess  task  criticality  (the 
latter  is  only  necessary  if  effectiveness  is  being 
calculated).  These  scores  must  be  then  entered  into  the 
P)TES  data  base  management  system,  the  System  Description 
Technology,  When  this  is  completed,  users  must  enter  the 
Applications  Program  mode  in  the  System  Description 
Technology,  and  select  the  Media  Selection  Program.  Once  in 
the  Media  Selection  Program,  users  must  then  select  criteria 
to  be  used  in  making  media  assignments.  Seven  options  for 
selecting  criteria  are  provided:  (1)  efficiency,  (2) 
"effectiveness"  (3)  relative  cost,  (4)  cost  and  efficiency, 
(5)  cost  at.d  "effectiveness,"  (6)  a  utility  measure, 
combining  efficiency  and  cost,  and  (7)  a  utility  measure 
combining  "effectiveness"  and  cost. 

Once  the  criteria  have  been  selected,  users  must  select  the 
tasks  to  be  included  in  the  analysis  (only  tasks  already  in 
tne  SDT  may  be  selected).  Typically,  the  tasks  for  single 
course  module  will  be  selected  for  each  analysis.  With  the 
analysis  criteria  identified,  the  psychological  variables  to 
be  used  in  calculating  the  match  between  tasks  and  media 
must  be  selected.  Users  may  select  from  12  variables 
assessing  stimuis  cha racter ist  ics ,  six  variables  assessing 
response  characteristics  and  t  jur  variaDles  assessing 
feedback  cha r a c ter i s t  i  cs  .  ^ 


These  psycho  I'.Dg  i  ci  1  variables  wer-^  taken  Oirectly  fr  mu 
TFF.M  Mo'iel  (Torjeson,  Kunula,  an-l  Atlas;  19^1) 
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If  the  user  has  selected  a  set  of  criteria  involving 
effectiveness  (effectiveness,  cost  and  effectiveness,  or 
utility  with  effectiveness),  the  weights  for  the  component 
criticality  variables  must  be  entered  so  that  a  composite 
task  criticality  score  can  be  computed.  Eight  variables  may 
be  used  in  the  calculation  of  task  criticality:  (task 
frequency,  percent  member  performing,  percent  time 
performing,  task  delay  tolerance,  consequences  of  inadequate 
performance,  task  learning  difficulty,  probability  of 
deficient  performance  and  time  between  entry  and 
performance ) . 

At  tnis  point,  the  program  has  all  of  the  information  needed 
to  calculate  efficiency  and/or  effectiveness.  It  will  use 
this  information  to  calculate  the  relevant  measure  and  it 
will  display  the  results. 

If  the  user  has  selected  one  of  the  two  utility  measures 
(combini-ng  either  cost  and  efficiency  or  cost  and 
effectiveness)  the  user  will  then  be  required  to  enter  the 
weights  to  be  used  in  the  computation  of  utility.  Following 
this,  the  user  must  select  the  objectives  and  constraints  to 
be  used  in  assigning  tasks  to  media.  A  listing  of  the 
possible  combinations  of  objectives  and  constraints  is 
displayed  in  Table  A-1.  Once  this  information  has  been  put 
into  the  computer,  the  program  will  optimally  assign  the 
tasks  to  training  media.  For  instance,  if  the  user  selected 
"maximum  effectiveness"  as  an  objective  and  "minimize  cost" 
as  a  constraint,  the  program  would  determine  the  assignment 
of  tasks  to  media  which  gives  the  highest  overall  score  on 
effectiveness  and  still  remains  under  a  user-specified  level 
of  overall  cost.  Once  the  initial  assignments  have  been 


examined,  ttie  user  can  examine  the  effects  of  several 
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ks  to  media  is  yenerated  sucti  tiiat  lotal 
nd  total  effectiveness  is  greater  than  or 
d  level  of  efficiency. 


alternatives  including  changes  in  (1)  objective,  (2) 
constraint,  (3)  criteria,  (4)  task  criticality  variables, 
(5)  psychological  variables,  (6)  task  criticality  variable 
weights,  and/or  (7)  utility  weights. 

After  users  have  explored  these  alternatives,  the  task-media 
assignments  can  be  finalized  and  entered  into  the  SDT. 


APPENDIX  B;  OVERVIEW  OF  AUTOMATED  RESOURCE  AND  COST 

ESTIMATION  TECHNIQUE 


This  appendix  provides  detailed  instructions  on  how  to  use 
the  Automated  Resource  and  Cost  Estimation  Technique 
(RCET).  A  more  detailed  description  of  RCET  is  contained  in 
the  User's  Guide:  Resource  and  Cost  Estimation  Technique. 
The  purpose  of  RCET  is  to  provide  Army  Training  analysts 
with  an  automated  tool  for  estimating  instructor 
requirements  and  institutional  training  costs  during  the 
earliest  phases  of  the  acquisition  process. 

The  Automated  Resource  and  Cost  Estimation  Technique  (RCET) 
is  designed  to  use  input  data  from  the  ETES  data  base 
management  system,  the  System  Description  Technology  (SDT). 
Actual  calculation  of  instructors  and  institutional  training 
course  cost  in  RCET  is  accomplished  by  using  the  VISICALC 
automated  worksheet  software  developed  by  Visicorp.  The 
VISICALC  worksheet  is  also  used  to  conduct  sensitivity 
analyses  of  key  parameters. 

B.l  CONCEPTUAL  OVERVIEW 

The  Resource  and  Cost  Estimation  Technique  has  three 
component  s : 

(1)  SDT  Interface  Software  -  this  software  is  used  to  select 
and  remove  data  from  the  SDT  and  to  format  the  data  for  use 
in  the  VISICALC  program.  In  addition,  it  is  used  to  copy 
the  results  of  the  VISICALC  program  back  into  the  SDT. 


12)  Tailored  VISICALC  Worksheet  -  this  worksheet  contains 
the  equations  for  determining  number  of  instructors  and 
course  costs.  In  addition,  it  contains  all  of  the  commands 
needed  to  load  and  unload  the  SDT  input  file,  and  to  conduct 
sensitivity  analyses.  This  tailored  worksheet  saves  the 
user  from  the  somewhat  tedious  process  of  setting  up  a 
VISICALC  worksheet  and  command  structure. 

(3)  Manual  Procedures  -  these  procedures  describe  how  to 
develop  input  data  and  how  to  use  the  SDT  interface  software 
and  the  tailored  VISICALC  worksheet. 

There  are  two  major  products  of  RCET:  (1)  a  listing  of  the 

number  of  instructors  required  in  the  course  and  (2)  a 
listing  of  projected  costs  for  the  course.  An  example  of 
the  cost  elements  estimated  by  RCET  is  listed  in  Table  B- 
1.  These  are  the  same  cost  elements  used  in  the  Cost 
Analysis  Program  of  the  Army  TRADOC  Resource  Management 
(ATRM)  system. 

B.1.1  Calculation  of  Course  Costs 

Costs  for  a  new  course  are  estimated  by  identifying  a 
comparable  existing  course,  obtaining  cost  data  from  this 
course  data  to  reflect  the  differences  in  key  resource 
requirements  (for  example,  number  of  students  and  number  of 
instructors)  between  the  comparable  course  and  the  new 
cou rse  . 

This  procedure  provides  estimates  of  course  costs  that  are 
(1)  empirically  based  and  (2)  suitable  for  the  types  of  high 
level  analyses  which  are  conducted  during  the  early  phases 
of  the  acquisition  process. 


Table  B-1  Example  Cost  Elements 
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B.1.2  Calculation  of  Number  of  Instructors 

The  number  of  instructors  required  in  a  course  is  calculated 
by  an  automated  version  of  the  algorithm  listed  in  the 
Staffing  Guide  for  U.S.  Army  Services  Schools  (DA  PAM  5  70- 
538)  . 

B.2  PROCEDURAL  OVERVIEW 

An  overview  of  the  procedures  in  the  Resource  and  Cost 
Estimation  Technique  is  provided  in  Figure  B-1. 

The  first  step  in  the  application  of  RCET  is  the 
identification  of  the  "reference"  course  or  the  comparable 
existing  cousre,  which  most  closely  resembles  the  task  and 
target  population  requirements  of  the  new  course. 
Procedures  for  identifying  a  reference  course  are  contained 
in  the  ETES  User  Guide  Manual  Procedures.  Once  the  reference 
course  has  been  identified,  cost  data  for  this  course  is 
obtained  from  the  Cost  Analysis  Program  (MOS  Training  Cost) 
and  entered  into  the  SDT. 

Reference  course  information  is  also  used  in  the 
construction  of  the  quasi-program  of  instruction  (QPOl)  for 
the  new  course.  Included  in  the  QPOI  is  a  description  of 
the  methods  to  be  used  in  each  module  in  the  course,  and  the 
student- instructor  ratio  and  instructor  contact  hours 
associated  with  each  method.  Procedures  for  constructing  a 
QPOI  are  contained  in  the  ETES  User's  Guide.  This  same 
guide  contains  procedures  for  determining  the  number  of 
students  to  be  trained.  This  value  is  a  critical  factor  in 
the  determination  of  course  costs. 


Once  the  QPOI  has  been  instructed  for  the  new  course, 
information  from  the  QPOI  on  instructional  methods,  student/ 
instructor  ratios,  and  contact  hours  must  be  entered  into 
the  SDT.  When  this  is  completea  the  the  user  must  enter  the 
SDT,  enter  the  Applications  mode,  select  the  Resource  and 
Cost  Estimation  Techniques  (RCET),  and  copy  the  reference 
cost  data  and  new  course  QPOI  data  onto  files  which  can  be 
read  into  the  VISICALC  program. 

Once  the  VISICALC  input  files  have  been  developed,  the  user 
must  remove  the  SDT  software  diskette,  put  in  the  VISICALC 
software  diskettes  and  enter  the  VISICALC  program.  Once  into 
the  VISICALC  software,  a  few  simple  commands  may  be  used  to 
load  the  SDT  input  files  and  RCET  worksheet  into  the 
VISICALC.  When  this  is  completed,  a  small  selected  set  of 
resource  data  for  the  reference  and  new  course  must  be 


entered  into  the  SDT. 


Course  costs  and  instructor 


requirements  may  then  be  calculated  by  executing  a  simple 
command  built  into  the  RCET  worksheet. 


After  examining  the  initial  estimates  of  course  costs  and 
instructor  requirements,  the  user  can  then  use  a  few 
commands  built  into  the  RCET  worksheet  to  conduct 
sensitivity  analyses  of  key  parameters.  When  these  analyses 
are  complete,  tiie  final  set  of  costs  for  the  new  course  can 
be  copied  onto  a  VISICALC  output  file.  The  user  can  then 
exit  the  VISICALC  software,  enter  the  SDT  software  and  copy 
the  output  file  into  the  SDT  data  base. 


APPENDIX  C:  OVERVIEW  OF  AUTOMATED  TRAINING  PLANNING 
AND  SCHEDULING  TECHNIQUE 


This  appendix  provides  an  overview  of  the  Automated 
Scheduliny  and  Planning  Techniques  (APST).  A  more  detailed 
description  of  APST  is  provided  in  the  User's  Guide: 
Automated  Planning  and  Scheduling  Technique.  APST  is 
designed  to  assist  training  developers  in  describing  and 
monitoring  the  training  development  schedule  for  developing 
Army  weapon  systems.  APST  is  designed  to  be  used  with  the 

VisiSchedule  software,  which  is  an  automated  program  for 
describing,  monitoring  and  reporting  schedule  information 
and  for  conducting  critical  path  analyses  of  schedule 

events.  A  data  input  diskette,  describing  the  events 

required  in  the  Army's  Individual  and  Collective  Training 
Plan  (ICTP),  is  included  in  these  techniques.  This  data 

input  diskette  contains  detailed  information  on  the 
sequential  relationships  among  the  events  in  the  OICTP. 

APST  is  designed  to  make  it  relatively  easy  for  training 

developers  to  track  and  monitor  the  complex  relationships 
among  the  events  in  the  training  development  schedule.  In 

addition,  by  providing  an  automated  capability  to  monitor 

tne  training  schedule,  it  should  aid  training  developers  in 

responding  quickly  and  efficiently  to  the  frequent  schedule 
ohdn(jes  which  occur  during  the  development  of  Army  weapon 
svs  terns . 


C.l  CONCEPTUAL  OVERVIEW 


Construction  of  training  development  schedules  for  emerging 
s/stems  is  a  difficult  task.  Over  100  developmental  events 
are  listed  in  TRADOC  Reg  351-9.  The  sequential 

relationships  among  these  events  are  complex  and  are  not 
described  in  any  systematic  and  integrated  manner  in  TRADOC 
Reg  351-9. 

Further,  the  training  scheduling  process,  particularly 
during  the  early  phases  of  system  development,  is 
cha racter  ized  by  frequent  changes  and  updates.  Determina¬ 
tion  of  the  impact  of  these  changes  is  a  tedious  and  time 
consuming  process. 

APST  contains  techniques  for  using  automated  VisiSchedule 
software  to  track  and  monitor  training  development  schedule. 
Hy  using  VisiSchedule,  the  training  developer  can  quickly 
and  efficiently  respond  to  changes  in  the  training 
developnient  scheudle.  Use  of  the  VisiSchedule  program  is 
facilitated  by  the  inclusion  of  an  input  data  diskette  which 
(a)  describes  the  events  in  the  training  development  process 
(as  specified  in  TRADOC  Reg  351-9),  (2)  describes  the 

temporal/sequent  la  1  relationships  among  these  events,  and 
(3)  lists  the  expected  duration  of  these  events  for  a 
"typical"  major  Army  weapon  system.  This  data  diskette 
significantly  reduces  data  input  requirements.  In  addition, 
it  eliminates  the  need  for  an  analysis  of  the  complex 
sequential  relationships  among  training  development  events 
which  are  either  implicitly  or  explicitly  specifo^d  in 
TRADOC  351-9. 
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Capabilities  of  VisiSchedule  Sof twa re 


As  applied  to  the  training  development  process,  the 
VisiSchedule  software  can  be  viewed  as  providing  the 
following  capabilities; 

(1)  Allows  users  to  systematically  describe  an 
integrated  training  development  schedule  including 
information  on  training  development  events,  the 
sequential  relationships  among  these  events,  the 
duration  of  these  events,  the  manpower  (by  labor 
category)  required  to  accomplish  each  event,  and 
the  costs  (that  is,  salaries)  of  this  manpower. 

(2)  Allows  users  to  quickly  determine  the  impact  of 
changes  to  any  of  the  above  information. 

(3)  Allows  users  to  identify  the  "critical  path"  in 
the  training  development  schedule.  A  "critical" 
event  is  one  whose  delay  would  impact  completion 
of  the  whole  project. 

(4)  Allows  user  to  aggregate  events  to  determine  total 
manpower  requirements  (by  paygrade  or  occupational 
specialty)  and  ho  determine  total  training 

-  development  costs. 

C.2  PROCEDURAL  OVERVIEW 

An  overview  of  the  procedures  for  using  the  automated 
planning  and  scheduling  techniques  is  provided  in  Figure  C- 


C-3 
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The  first  procedure  involves  setting  up  the  hardware  and 
software  needed  to  run  the  automated  techniques.  As  part  of 
this  procedure,  the  VisiSchedule  software  and  the 
accompanying  input  data  diskette,  describing  the  ICTP  events 
and  their  interrelationships  are  entered  into  the  computer. 

In  the  next  three  procedures  (2.1  and  2.3),  the  data  on  the 
input  data  diskette  describing  the  ICTP  events,  their 
interrelationships,  and  their  durations  is  reviewed  by  the 
user.  If  the  user  feels  that  the  existing  data  is 
acceptable  and,  thus,  is  an  accurate  description  of  his/her 
training  development  schedule,  the  user  can  proceed  directly 
to  the  output  report  procedure  (4.0).  However,  it  is  more 
likely  that  the  user  will  want  to  change  the  durations  or 
completion  deadlines  of  some  of  the  events;  the  events 
themselves  and  their  relationships  are  less  likely  to 
require  change. 

If  changes  are  required,  these  changes  may  be  made  using  the 
methods  described  in  procedures  3.1  to  3.4.  Procedure  3.4 
allows  users  to  enter  and/or  modify  data  on  manpower 
requirement  and  costs.  Information  on  manpower  requirements 
must  be  entered  by  the  user  since  this  information  is  too 
system  specific  to  include  in  the  input  data  diskette. 

In  procedure  4.0,  the  user  may  select  from  one  of  four 
different  output  reports  to  describe  the  training 
development  schedule.  After  examining  these  outputs,  the 
user  may  wish  to  conduct  tradeoff  analyses  or  sensitivity 
analyses  of  the  schedule  input  variables.  This  can  be 
accomplished  by  changing  the  input  parameters  through 
orocedures  2,0  and  3.0. 


APPENDIX  D:  DESCRIPTION  OF  PROJECTS  RELATED  TO  ETES 


This  appendix  provides  descriptions  of  the  six  ARI  projects 
which  are  directly  related  to  ETES:  (1)  the  Hardware 
Procurement-Military  Manpower  (HARDMAN)  Methodology,  (2)  the 
Man  Integrated  System  Technology  (MIST),  (3)  the  Training 
Efficiency  Estimation  Model  (TEEM),  (4)  the  Training 
Developer's  Decision  Aid  (TDDA)  ,  (5)  the  Model  for  the 
prediction  of  the  effectiveness  of  Training  Devices 
(TRAINVICE),  and  (6)  the  Army  Manpower  and  Personnel 
Requirements  Process  (ARMPREP).  More  details  on  these  six 
projects  are  provided  in  the  subsections  which  follow. 


D.l  MILITARY  MANPOWER  VERSUS  HARDWARE  PROCUREMENT  (HARDMAN) 

The  HARDMAN  methodologies  provide  the  Navy  and  the  Army  with 
a  systematic  means  of  estimating  human  resource  requirements 
(manpower,  personnel,  and  training)  during  the  earliest 
phases  of  the  Weapon  System  Acquisition  Process  (WSAP).  The 
integrated  models  and  data  bases  of  the  methodologies  are 
designed  to  (1)  assess  manpower,  personnel,  and  training 
requirements  for  proposed  sytems;  (2)  determine  the  impacts 
of  manpower,  personnel,  and  training  requirements;  and  (3) 
identify  and  evalute  tradeoffs  which  would  alleviate 
unfavorable  impacts.  DRC  has  also  extended  the  capability 
of  HARDMAN  through  increased  automation  and  the  development 
of  proprietary  software  packages.  These  advances  have 
significantly  cut  cost  and  response  time  and  more  closely 
aligned  the  output  products  of  HARDMAN  with  the  information 
needs  of  Program  Managers,  resource  sponsors  and  their 


In  its  present  form/  DRC's  HARDMAN  Methodology  is  composed 
of  six  interrelated  steps  which  can  be  iterated  in  a  timely 
fashion  for  the  analysis  of  alternative  design  proposals. 
The  first  step  is  conerned  with  various  aspects  of  data 
collection,  generation,  formatting,  and  analysis.  It  also 
includes  the  establishment  of  an  automated  audit  trail  or 
program  record  of  all  input  data,  analyses  and  output 
products.  The  final  five  steps  involve  data  evaluation  with 
respect  to  acquisition  program  goals  and  constraints. 

The  establishment  of  the  consolidated  data  base  triggers  the 
methodology  and  ensures  that  all  analyses  pertaining  to  a 
new  weapon  system  use  a  common  data  source.  Consequently, 
consistent  definitions  and  collection  procedures  are 
employed  in  assembling  data  on  operator  and  maintainer 
functions,  the  weapon's  operational  mission/scenario  and 
maintenance  concept,  its  cost  factors,  and  its  manpower  and 
training  sytem  support  requirements.  The  proposed  weapon 
system's  demand,  in  terms  of  manpower,  personnel,  and 
training  is  determined  in  Steps  2  through  4  and  compared  to 
the  projected  supply  of  such  resources  at  system  Initial 
Operational  Capability  (IOC)  in  Step  5.  Significant 
shortfalls  and  equipment  sources  of  high  resource  demand  are 
identified.  In  Step  6,  the  methodology  is  iterated  to 
examine  alternative  designs,  training  methods  and  media,  and 
total  force  tradeoffs.  Thus  the  program  manager  can 
participate  in  the  design  process  with  a  heightened 
awareness  of  the  training  and  manpower  demands  of  a  new 
weapon  system,  thereby  ensuring  its  supportabi 1 i ty  as  well 
as  its  mission  capability. 


DRC's  HARDMAN  Methodology  has  been  used  on  a  wide  range  of 
all  three  Military  Services,  including  the  Shipboard 
Intermediate  Range  Combat  (SIRCS);  the  lSD-41,  a  new  class 
of  guided  missile  destroyer  (DDG-51);  the  Submarine  Advanced 
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I  Combat  System  (SUBACS);  the  Army's  Divison  Support  Weapon 

System  ( DSWS ) ;  the  Corps  Support  Weapon  System  (CSWS);  and 
the  Remotely  Piloted  Vehicle  (RPV);  and  in  modified  form 
with  the  Air  Force's  Combat  Identification  System  (CIS). 

These  applications  have  fully  exercised  all  of  the  analytic 
capabilities  of  the  methodology.  Additionally/  they  have 
demonstrated  its  capability  to  provide  timely,  accurate 
support  for  program  management  and  review.  Specifically, 
significant  or  problematic  human  resource  demand  generated 
by  equipment  design  has  been  targeted  early  in  program 
development;  alternative  system  and  subsystem  configurations 
have  been  evaluated  through  tradeoff  studies;  and  the 
methodology's  consolidated  data  base,  supported  by  carefully 
documented  resource  design  analyses,  has  been  used  to  build 
an  audit  trail  of  manpower,  personnel,  and  training  assess¬ 
ments  and  tradeoffs. 


D.2  MAN  INTEGRATED  SYSTEMS  TECHNOLOGY  (MIST) 

The  United  States  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  (ARI)  is  developing  a  technology  which 
integrates  manpower,  personnel  and  training  (MPT)  consider¬ 
ations  throughout  the  Weapon  System  Acquisition  Process 
(WSAP).  Its  goal  is  to  ensure  the  effective  planning  for 
and  utilization  of  projected  human  resources  for  operational 
readiness.  The  technical  activity  required  to  achieve  this 
objective  has  been  organized  into  a  major  research  and 
development  program  entitled  Man  Integrated  Systems 
Technology  (MIST).  DRC  has  been  selected  as  the  contractor 
for  MIST  which  is  an  extensive  (5  year,  32  man-years)  multi- 
million  dollar  effort. 


MIST  Will  consist  of  the  necessary  technology  and  management 
procedures  to  address  the  following  considerations:  (1)  the 
treatment  of  human  resources  as  a  performance  and  cost 
factor  during  system  concept  formulation,  (2)  the  planning 
and  forecasting  of  manpower  information,  (3)  the  parallel 
development  of  associated  training  systems  with  weapon 
systems,  and  finally  (4)  the  specification  of  test  and 
evaluation  issues  to  ensure  human  resource  accountability, 
and  to  support  the  ASARC/DSAftC  review  at  each  of  the  major 
development  milestones.  MIST  in  particular  will  integrate 
and  demonstrate  the  technical  relationships  among  these 
considerations,  and  provide  the  necessary  methodology  to 
ensure  their  effective  treatment  during  the  design  process 
frcxn  a  performance,  manpower,  training  and  cost  point  of 
view.  In  addition,  MIST  will  demonstrate  how  these 

technical  cons idera t ions  are  related  and  responsive  to  the 
Army's  Life  Cycle  System  Management  Model  (LCSMM),  and  to 
relevant  Army  regulations  and  responsible  agencies.  MIST 
will  further  provide  a  comprehensive  framework  for  the 
assessment  and  integration  of  other  relative  technologies 
and  information  as  they  are  identified  or  developed. 


When  fully  developed,  MIST  will  serve  as  an  integrating 
mechanism  providing  efficient  interfaces  for  information 
exchange  and  feedback  among  concurrent  activities  and/or 
processes  involving  industry  and  government,  system  design 
engineers  and  support  planners.  Army  acquisition  managers, 
and  the  Army's  oversight  bureaucracies  in  the  Department  of 
Defense  and  the  Congress.  Hence,  MIST  will  provide  the 
design  and  implementation  of  more  comprehensive  and 
responsive  data  bases  and  technical/management  procedures 
employed  in  prime  and  support  system  development, 
acquisition  and  operation. 


When  completed,  MIST  will  be  composed  of  three  principal 
components ; 

o  Analytical  Tool  Designed  to  Support  Program 

Managers,  TRADOC  System  Managers,  Acquisition 
Managers  (e.g..  Weapon  System  Manager  (WSM)  in 
DARCOM),  and  their  staffs/supporting  agencies; 

o  A  system-specific  data  base  to  provide  a  single 

point  source  for  all  man-machine  data  (input  to 
the  tools,  output  from  them  and  decision  support 
information  for  program  documentation  and  reviews) 
collected  for  an  emerging  system. 

o  A  management  information  system  to  link  MIST  users 
and  the  data  base  in  an  effective  communication 
network  within  the  framework  of  the  Life  Cycle 
System  Management  Model. 

Hence,  MIST  will  provide  an  important  new  capability  within 
the  Army.  This  capability  is  embodied  in  a  cost-and  time- 
efficient  system  for  human  resource  requirements  analysis 
early  in  weapon  system  development,  MIST  will  capture  the 
most  recent  advances  in  both  man-machine  technology  and 
automated  data  processing  to  provide  a  system  which  is  user- 
friendly  at  the  operational  level  and  responsive  to  the 
analysis  and  information  requirements  of  users  at  several 
levels  within  the  chain  of  command.  Finally  MIST  will  "fit 
into  the  real  world"  by  providing  effective  interfaces  to 
existing  management  information  systems  and  data  bases  for 
logistic  support  analysis,  training  system  development  and 
personnel  planning  in  DARCOM,  TRADOC  and  the  Army  Staff. 


D.3  TRAINING  EFFICIENCY  ESTIMATION  MODEL  (TEEM) 


TEEM  is  an  interactive  computer-based  aid  designed  to  assist 
training  developers  in  early  evaluation  and  generation  of 
training  program  components.  At  the  foundation  of  TEEM  is  a 
set  of  psychological  variables  which  reflect  the  training 
information  contained  in  a  system  task  description.  This  is 
in  contrast  to  the  normal  textual  form  in  which  task 
descriptions  are  usually  developed.  A  representation  by 
variables  was  chosen  because  early  task  descriptions  are 
often  not  in  standard  formats,  may  change  quickly,  and  may 
require  elaboration  by  design  engineers  or  other  subject 
matter  experts.  A  predetermined  set  of  standard  psycho¬ 
logical  variables  provided  a  starting  point  which  could  be 
depended  upon  for  input  into  the  fixed  operational  code  of  a 
computer  program. 

Initial  task  groups  corresponding  to  procedures  that  should 
be  trained  together  are  generated  through  an  algorithm  that 
clusters  tasks  based  upon  internal  similarity  along  the 
psychological  variable  dimensions.  After  clustering  is 
complete,  a  task  group  has  maximum  difference  between  task 
clusters.  The  clusters  serve  as  an  automated  estimate  for 
use  during  the  initial  formulation  of  a  training  program. 
Task  groups  are  then  modified,  if  needed,  based  on  the 
expertise  of  the  analyst  or  outside  mitigating  factors  such 
as  management  and  cost  constraints  that  would  normally  not 
be  included  in  the  psychological  variables. 

Selection  of  concrete,  costable  components  such  as  media  and 
method  of  training  are  based  upon  the  similarity  of  a  group 
of  task  variable  requirements  to  the  ability  of  a  media  or 
method  to  manipulate  psychological  variables  during 
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training.  Selection  is  accomplished  by  representing  each 
training  program  component  by  a  description  in  the  same  set 
of  psychological  variables.  A  task  is  also  coded  as  a 
vector  of  variables  with  an  assocated  variable  value  for 
each  dimension  representing  applicability  or  non¬ 
applicability.  Selection  of  a  program  component  is 

performed  by  automatically  matching  each  task  to  each 
potential  component  and  selecting  the  component  with  the 
higher  match  of  critical  variables. 

If  an  already  existing  training  component  is  being 
evaluated,  the  matching  process  is  utilized  in  reverse  to 
generate  a  figure  of  aggreement  between  the  theoretical 
maximum  possible  under  the  task  and  hardware  conditions  and 
the  actual  match  under  a  particular  alternative.  The 
normalized  numeric  value  of  this  match  is  called  the 
efficiency  ratio. 

After  efficiency  ratios  have  been  generated  for  potential 
training  programs  a  final  selection  for  the  best  training 
approach  is  made  based  on  pair  values  of  gross  dollar  costs 
and  their  efficiency  ratios.  Together  the  values  represent 
an  early  form  of  cost-effectiveness  ratio  for  each  approach. 
The  best  candidate  programs  are  selected  for  detailed  cost 
analysis  through  a  much  more  complex  accounting  program  and 
given  increased  program  specification. 

TEEM  was  constructed  with  the  realities  of  the  early 
training  situation  in  mind.  It  does  not  assume  extensive 
and  accurate  information.  It  is  "user  friendly"  with  step- 
by-step  guidance  to  individuals  unfamiliar  with  micro- 
ccjmputers.  It  is  designed  to  accommodate  local  school  needs 
and  training  development  procedures  without  major  program 


changes.  It  is  also  flexible  enough  to  be  quickly  modified 
based  upon  improvements  in  psychological  research  and  field 
experience . 

D.4  TRAINING  DEVELOPER'S  DECISION  AID  (TDDA)* 

The  Training  Developer's  Decision  Aid  (TDDA)  was  designed  to 
assist  training  development  specialists  in  applying  the 
Instructional  Systems  Design  (ISD)  process.  The  ISD  process 
is  a  comprehensive  technique  for  the  systems  a  ^oach  to 
development  training.  Applied  Science  Associ  "^s,  Inc. 
(ASA),  working  in  collaboration  with  the  Arr  Research 
Institute  Field  Unit  at  Fort  Bliss,  Texas,  has  d  aloped, 
tested,  revised,  computerized,  and  conducted  tryouts  of  the 
model  at  five  Army  schools. 

The  TDDA  is  intended  to  be  an  aid  to  training  developers  who 
need  technical  assistance  in  designing  or  redesigning 
training  and  to  streamline  the  process  of  making  training 
decisions.  The  TDDA  is  procedural  in  nature;  it  leads 
training  developers  through  a  series  of  procedures  which 
specify:  (1)  what  decisions  are  to  be  made,  (2)  a  rational 
order  for  decisions,  and  (3)  the  nature  and  sources  of 
information  required  to  make  valid  training  decisions.  The 
model  also  serves  to  organize  and  retain  information  that  is 
gathered  and  processed  during  the  course  development 
process . 


i'he  TDDS  model  can  be  used  with  partial  or  complete  task 

lists  or  performance  objectives.  It  is  not  necessary  to 

redo  the  job  analysis  in  order  to  use  the  TDDA  process. 

This  reflects  a  training  development  philosophy  that  a 

thorough  analysis  of  the  job  early  in  the  training 

development  process  is  essential  for  making  valid  training 
development  decisions. 

The  Job  Analysis  Module  requires  the  greatest  input  form 
subject  matter  experts  (SMEs).  Those  who  hold  the  MOS  being 
analyzed  and  selected  to  serve  as  SMEs  are  expected  to  be 
expert  job  performers  with  at  least  eighteen  months  of 
recent  on-the-job  experience  at  an  operational  site.  This 
is  an  essential  requirement  for  the  successful  application 
of  the  model.  The  model  is  sensitive  to  quality  of  the 

input  data  and  information,  since  all  three  major  decision 
(what,  where  and  how  to  train)  are  directly  tied  to  input. 

There  are  two  separate  parts  to  the  SME  input  in  the  Job 

Analysis  Module.  First,  the  initial  task  list  is  generated 
by  exercising  function  analyzers.  Then,  the  task  list  is 
verified  as  correct  and  complete.  This  should  be  carried 
out  by  a  second  set  of  SMEs.  At  the  time  the  tasks  are 
verified,  the  remainder  of  the  input  information  is  obtained 
from  the  SMEs.  The  output  of  this  module  provides  the 
inswer  to  the  question,  "What  should  be  trained  to  prepare 

someone  to  perform  in  their  MOS?" 


The  Functional  Learning  Requirements  Module  requires  inputs 
from  tw(3  sources,  MOS  SMEs  and  course  development  instructor 
^jersonnel  in  order  to  tnake  decisions  of  how  to  train.  The 
information  generated  in  this  module  is  used  by  the  coarse 
‘ieveloper  to  specify  the  functional  characteristics  of  tne 


training  program.  The  emphases  in  specifying  these 
characteristics  are  on  how  best  to  communicate  the  learning 
objectives,  job  context  and  consequences  of  task 
performance,  and  on  providing  for  efficient  practice  for 
acquiring  special  job  skills. 

Development  of  the  course  requires  that  the  rela t ionships 
between  tasks  be  determined.  The  last  SME  input  provides 
the  information  for  describing  these  relationships.  Tasks 
are  related  in  three  ways.  First,  the  same  skills  may  be 
required  in  the  performance  of  different  tasks.  Knowing 
which  skills  are  redundant  allows  for  more  efficient 
scheduling  and  sequencing  of  instructional  elements. 
Second,  some  tasks  must  be  performed  on  the  job  before  other 
tasks  can  be  performed.  For  example,  a  piece  of  equipment 
cannot  be  repaired  before  the  cause  of  the  malfunction  is 
isolated.  Or,  an  intelligence  report  cannot  be  prepared 
before  the  intelligence  information  and  data  are  analyzed. 
The  output  of  these  tasks  are  inputs  to  the  subsequent 
tasks.  The  third  relationship  is  merely  chronological.  One 
task  must  be  performed  prior  to  a  second  one  but  there  is 
not  direct  output-input  dependency.  For  example,  a  weapon 
system  has  to  be  emplaced  (a  collective  task)  and  made  ready 
for  action,  before  it  can  be  used  to  engage  hostile  targets, 
which  may  or  may  not  occur. 

The  dependency  relationship  information  is  provided  by  SMEs 
and  is  then  used  to  build  a  task  dependency  hierarchy.  The 
product  is  considered  to  be  a  general  course  structure  upon 
which  the  course  map  can  be  built.  The  task  dependency 
information  is  generated  in  the  Functional  Learning 
Requirements  Module,  but  is  analyzed  in  the  Structure 
Designation  Module. 
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Information  and  data  from  several  earlier  inputs  are  used  in 
the  last  module  to  ,?ke  decisions  as  to  where  the  training 
should  be  conducted.  The  task  criticality  data  is  one  of 
the  primary  inputs  to  this  decision.  Additionally,  the 
instructional  setting  characteristics  and  school  context 
information  are  used  in  reaching  these  conclusions. 

The  TDDS  model  then  is  used  to  make  three  major  decisions: 
what  to  train,  how  to  train,  and  where  to  train.  Some 
decisions  are  made  algorithmically  and  others  are 
heuristically  determined. 


D.5  A  MODEL  FOR  THE  PREDICTION  OF  THE  EFFECTIVENESS  OF 
TRAINING  DEVICES* 

TRAINVICE  is  a  methodology  for  the  systematic  assessment  of 
the  characteristics  of  training  devices  under  development. 
The  model  is  based  on  the  assumption  that  certain  attributes 
to  be  assessed  in  the  training  situation  will  lead  to 
transfer  of  training  to  the  operational  situation.  There¬ 
fore,  the  higher  the  rating  on  the  assessment  factors,  the 
higher  the  transfer  that  will  take  place  and  the  more 
effective  the  device.  The  model  provides  a  framework  for 
the  making  of  these  judgments.  The  three  variables  entering 
into  the  assessment  are:  (1)  the  transfer  potential  of  the 

device,  (2)  the  learning  deficit  to  oe  overcome  and  (3) 
instructional  effectiveness.  As  with  any  model,  its 
effectiveness  depends  on  the  adequacy  of  the  input  da  ta . 
Inputs  into  the  model  consists  of  descriptions  of  tasks  and 
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subtasks  represented  in  the  operational  situation,  as 
circumscribed  by  the  training  objective,  and  those 
represented  in  the  training  device.  The  controls  and 
displays  and  their  functions  for  both  situations  are  listed. 
In  addition,  the  skills  and  knowledges  involved  in  each 
subtask  in  the  operational  situation  are  formulated  for  use 
in  the  model.  Using  these  inputs,  judgments  are  made  using 
rating  scales.  The  subtasks  in  the  two  situations, 
operational  and  training,  are  compared  to  ascertain  if 
provision  is  made  for  representation  of  the  subtasks  in  the 
training  device  in  the  commonality  analysis.  Next,  the 
displays  and  the  controls  for  both  situations  are  compared 
on  physical  and  functional  similarity.  The  more  similar  the 
display  or  control  in  the  training  device  is  to  the 
operational  situation,  the  higher  the  score.  This  is  based 
on  the  premise  that  the  greater  the  physical  or  functional 
similarity,  the  greater  the  transfer  of  training  that  will 
result.  Physical  similarity  refers  to  the  appearance  and 
physical  aspects  of  the  displays  and  controls  involved; 
i.e.,  their  "fidelity";  functional  similarity  involved  in 
the  operation  of  control,  in  information  processing  terms. 
The  learning  deficit  analysis  is  based  upon  (1)  the  assess¬ 
ment  of  the  level  of  proficiency  in  each  skill  or  knowledge 
for  the  students  upon  entering  the  training  situation,  (2) 
the  desired  level  of  proficiency  in  each  skill  or  knowledge 
for  the  students  upon  leaving  the  training  situation,  and 
(3)  tne  difficulty  (in  terms  of  training  time)  of  training 
in  the  skills  or  knowledges  involved  in  a  subtask.  This 
analysis  yields  a  weighted  learning  deficit  for  each 
subtask.  The  judgments  concerning  the  level  of  each  skill 
or  knowledge  are  made  using  scales  adapted  from  Demaree 
(1961).  The  last  analysis  involved  in  the  TRAINVICE  model 
is  an  assessment  of  how  well  the  training  device  adhers  to 
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"good"  training  techniques.  In  order  to  perform  this 
analysis,  each  of  the  subtasks  is  cast  into  one  or  more 
categories  of  behavior.  These  categories  are  those  of 
Braby,  et  al  (1972),  which  are  derived  for  an  earlier 
behavioral  categorization  by  Willis  and  Peterson  (1961). 
For  each  of  the  behavioral  categories  represented  in  the 
subtask,  a  list  of  guidelines,  also  those  of  Braby,  et  al. 
(  1972),  are  consulted  and  judgments  made  of  the  degree  to 
which  the  guidelines  are  followed,  or  not  followed,  relative 
to  the  manner  in  which  the  subtask  is  represented  in  the 
training  device.  The  guidelines  are  broken  up  into  those 
dealing  with  the  stimulus,  response,  and  feedback  aspects  of 
the  training  situation.  For  each  subtask,  the  lowest 
obtained  score  on  each  of  the  three  aspects  is  used  to 
derive  an  average  training  technique  score.  All  of  the 
preceding  rating,  are  then  fed  into  an  equation  to  formulate 
an  index  of  prediction  of  training  effectiveness,  ranging 
from  0  to  1.  This  equation  is  as  follows: 


(Ci  X  Si  X  Ti  X  Di) 


where  C  is  task  commonality,  S  similarity,  T  training 
techniques,  and  D  the  training  deficit  scores  for  each 
subtask.  The  equation  was  derived  from  a  transfer  of 
training  equation  of  Gagne,  Foster  and  Crowley  (1943),  which 
was  for  use  with  empirical  data,  while  the  TRAINVICE  extra¬ 
polation  deals  with  judgments  made  concerning  aspects  of  a 
device  assumed  to  bring  about  subsequent  transfer  of 
tra ining. 


A  validation  study  has  been  performed  on  the  model/ 
utilizing  data  obtained  during  the  course  of  two  field 
studies  as  criteria  against  which  to  compare  the  predictions 
derived  from  use  of  the  model  (Wheaton,  et  al.,  1976).  The 
devices  were  tank  gunnery  trainers  involved  with  burst-on- 
target  techniques  and  tracking  with  the  main  gun  of  the 
M60A1  tank.  In  each  case,  the  prediction  of  no  differences 
between  the  training  devices  involved  was  found  to  be 
consistent  with  the  equivalence  in  transfer  actually  found 
in  utilization  of  the  various  devices.  This  was  felt  to  be 
a  promising  but  not  definitive  finding. 

In  order  to  obtain  additional  validation  data  on  the  model, 
and  also  to  obtain  experience  in  utilization  of  the  model  to 
determine  if  there  were  aspects  that  might  be  changed  in 
order  to  enhance  the  practicality  of  utilization  of  the 
model,  the  Army  Research  Institute  personnel  applied  the 
model  to  two  maintenance  trainers  undergoing  evaluation  at 
the  Army  Ordnance  Center  and  School.  This  afforded  the 
opportunity  to  obtain  data  within  a  different  context  than 
that  dealt  with  by  gunnery  trainers. 

These  trainers  were  concerned  with  automative  trouble¬ 
shooting.  No  difference  in  training  effectiveness  was 
predicted  for  the  two  trainers,  which  agreed  with  the 
results  of  the  empirical  evaluation.  Various  aspects  of  the 
model  which  caused  difficulty  in  its  utilization  were  noted 
and  influenced  the  development  of  the  modified  version  .  In 
addition,  ARI  conducted  a  three-day  workshop,  in  which  the 
developers  of  the  original  model  and  individuals  who  had 
utilized  the  model  or  had  an  interest  in  its  utilization 
participated;  this  furnished  further  ideas  for  possible 
modi f ica  t ion . 
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D.6  ARMY  MANPOWER  AND  PERSONNEL  REQUIREMENTS  PROCESS 
(ARMPREP) 

The  Systems  Manning  Technical  Area  of  the  Systems  Research 
Laboratory,  US  Army  Research  Institute,  is  promoting  a 
research  thrust  known  as  Manned  System  Integration.  One 
major  component  of  this  program  is  the  development  of 
improved  procedures  for  the  manpower  planning  process.  The 
major  objective  of  this  proposed  research  is  the  development 
of  new,  innovative  and  effective  techniques  to  support  the 
determination  of  manpower  and  personnel  requirements. 
ARMPREP  will  include:  methodologies  to  accurately  estimate 
manpower  quantity  and  skill  level  of  personnel  through 
derivation  of  behavioral  requirements  and  subsequent 
translation  into  Military  Occupational  Specialty  (MOS)  and 
other  relationships;  techniques  to  aggregate  manpower  demand 
of  new  systems  for  comparison  with  available  supply; 
requirements  and  attributes  of  a  manpower  requirements 
management  information  system;  and  finally,  the  specifica¬ 
tion  for  a  computer  interactive  system.  ARMPREP  is  intended 
to  provide  tools  for  potential  integration  into  the  current 
manpower  planning  process  by  cognizant  Army  agencies.  In 
essence,  ARMPREP  has  four  principal  components  each  with  its 
own  identifiable  research  products.  These  four  components 
are ; 


o  Manpower  and  Personnel  Requirements  Determination 
Methodologies  (MANPERS). 


o 


Aggregation  Procedures  of  Manpower  Demand  (TOTAL 
MANPERS ) . 


o 


Requirements  for  a  Manpower  Requirements 
Management  Information  System  (MARMIS). 

o  Computer  Interactive  System  for  determination  of 
Manpower  and  Personnel  Requirements  {AUTO 
MANPERS) . 

MANPERS 

The  MANPERS  component  will  provide  the  techniques  and 
methodologies  to  formally  quantify  manpower  and  personnel 
requirements.  These  formal  methodologies  will  aid  in  the 
timely  and  accurate  determination  of  manpower  and  personnel 
requirements.  In  addition,  these  methodologies  are  intended 
to  standardize  the  manpower  and  personnel  requirements 
determination  process  to  provide  information  of  a  consistent 
nature  of  use  to  Army  personnel  subsystem  developers. 
Specific  aspects  of  the  MANPERS  component  will  consist  of 
the  following  attributes: 

(a)  The  development  of  a  taxonomic  model  to  derive 
behavioral  requirements  based  upon  new  system  Task 
Descriptive  Data  (TDD)  and  translation  of  these 
behavioral  requirements  into  MOS  and  associated 
decisions.  The  model  will  define  the  requisite 
data  to  be  included.  This  input  data  will  be 
obtained  from  sources  such  as  LSA,  Task  and  Skill 
Analyses  (TASA),  system  requirements  and  doctrine 
documents,  engineering  design  data  and  training 
plans.  The  model  will  specify  the  level  of 
specificity  of  da ta  congruent  with  each  phase  of 
the  LCSMM.  In  addition,  the  model  will  identify 
the  process  by  which  the  behavioral  requirements 


are  derived  (i.e.,  the  taxonomy)  and  the  means  by 
which  the  level  of  data  specifity  is  embraced  by 
the  taxonomy.  Finally,  the  MANPERS  methodologies 
will  guide  the  selection  of  personnel  for  a  given 
job  in  a  new  system  based  upon  the  behavior  (or 
performance)  expected  of  him,  as  opposed  to  the 
current  method  which  involves  the  preselection  of 
MOS  category,  previous  to  and/or  without  rigorous 
documentation  to  support  such  a  choice. 

(b)  Integration  with  events  and  activities  of  the 
LCSMM,  ILS  (LSA)  and  ISO.  The  MANPERS  process 
will  provide  the  necessary  manpower  input  to 
decision  makers  at  critical  system  development 
milestones  to  encourage  optimization  of  manpower 
planning.  Where  possible  it  will  support  the 
determination  of  manpower  related  life  cycle  and 
support  costs  relative  to  specific  systems. 
Through  involvement  with  the  ISD  model,  the 
MANPERS  component  will  support  a  margin  of 
requirements  for  new  equipment  training  (of  the 
Material  Developer)  with  training  requirements  (of 
the  Combat  Developer)  as  identified  by  specific 
schools.  Finally,  the  MANPERS  component  will 
serve  to  provide  increased  accuracy  of  information 
used  in  development  of  draft  plan  TOEs  (as  well  as 
throughout  the  TOE  planning  process). 

(c)  A  final  aspect  of  the  MANPERS  component  is  the 

development  of  a  MANPERS  manual  to  include:  job 

aids,  examples,  procedural  models,  instrument 
formats  and  presentation  methods  for  thorough 
manpower  and  personnel  requirements  determination. 
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The  MANPERS  manual  will  provide  a  tool  for  quick 
responses  to  needs  to  generate  quality  information 
on  a  timely  and  economical  basis.  The  MANPERS 
manual  will  facilitate  the  standardized  determin¬ 
ation  and  presentaiton  of  requirements  to  increase 
the  utilization  of  information  by  decision  makers. 
Finally,  the  MANPERS  manual  will  provide  a 

competent  training  tool  to  producers  of  manpower 

and  personnel  requirements  (especially  for  those 
new  to  the  job  and  content  area). 

TOTAL  MANPERS 

The  TOTAL  MANPERS  component  will  provide  the  techniques  for 
aggregation  of  manpower  demand  for  comparison  with  available 
supply  data.  Manpower  demand  is  defined  as  quantity  of 

manpower  by  quality  of  personnel  (e.g.,  skill  level)  both 

within  and  across  systems.  In  short,  it  is  a  quantity  by 
quality  of  manpower  comparison  between  demand  and  supply. 

Specific  aspects  of  the  TOTAL  MANPERS  component  include; 

(a)  Development  of  technqies  which  can  provide 
quantity  by  quality  loadings  of  manpower  demand 
for  a  specific  system.  The  data  used  by  TOTAL 
MANPERS  procedures  will  be  an  extension  of  the 
basic  MANPERS  tools  by  relating  some  paradigm  of 
personnel  performance  or  behavioral  requirements. 


Another  aspect  of  this  component  will  be  to 
provide  techniques  which  aggregate  information  for 
quantity  by  quality  of  manpower  across  systems 
through  comparison  of  manpower  demand  loadings 


held  in  common  with  personnel  performance  or 
behavioral  requirements. 

(c)  A  final  aspect  of  the  TOTAL  MANPERS  component  will 
be  its  coordination  with  events  and  activities  of 
the  LCSMM,  the  PPBS  (through  SACS  and  TOE 
planning)  and  ILS  to  insure  the  timely  determina¬ 
tion  of  requisite  manpower  information,  and 
authorizations,  as  well  as  changes  to  force 
structure  and  composition. 

MARMIS 

The  MARMIS  component  will  provide  the  manpower  management 
procedures  to  integrate  the  MANPERS  and  TOTAL  MANPERS 
components  with  the  Army  manpower  related  documentation 
process.  In  addition,  current  manpower  management  and 
documentation  procedural  deficiencies  and  recommended  fixes 
will  be  identified.  The  most  significant  activity  in  this 
component  is  to  support  current  Army  actions  in  management 
information  systems  developments  relating  to  the  manpower 
planning  process  through  the  specification  of  the  require¬ 
ments  and  attributes  of  MARMIS.  Specific  aspects  of  the 
MARMIS  component  include: 

(a)  Determination  of  manpower  management  and 

documentation  procedures  to  facilitate  the 

utilization  ot  manpower  and  personnel  requirements 
methodologies  and  aggregation  techniqes  so  as  to 
improve  the  timeliness  and  quality  of  requirements 


(b)  Identification  of  means  to  improve  accountability 
of  manpower  and  personnel  requirements  documenta¬ 
tion  to  identify  potential  economies  in  management 


and  documentation  flows. 


This  will  take  into 


account  organizational  and  technical  complexities 
which  serve  to  work  against  desired  results.  In 
addition,  these  efforts  will  facilitate  improved 
manpower  planning  as  well  as  to  focus  upon 
deficiencies  for  possible  resolution. 

(c)  Means  to  control  manpower  related  information  will 
be  developed  so  that  many  activities  concerned 
with  other  interests  (e.g.,  LCSMM,  SACS  and  TOE 
planning,  and  ILS)  can  be  fully  coordinated  with, 
such  that  appropriate  milestone  are  achieved  on  a 
timely  basis  and  the  requirements  of  other 

activities  are  met  as  they  relate  to  manpower 

concerns . 

AUTO  MANPERS 

The  AUTO  MANPERS  component  will  provide  the  basis  for 

development  of  a  computer  interactive  system  for  determining 
manpower  and  personnel  requirements  by  developing  the 

requirements,  attributes  and  specification  for  AUTO  MANPERS. 
AUTO  MANPERS  is  intended  to  provide  a  basis  for  MANPERS  an 
TOTAL  MANPERS  methodologies.  Specific  aspects  of  the  AUTO 
MANPERS  component  will  support  development  of  a  proposed 
computer  interactive  system  by  addressing  the  following; 


(a)  Ease  of  documentation,  update  and  edit  of  require¬ 
ments  through  use  of  an  online,  interactive  ADP 
system  which  provides:  continuous  availability  to 


authorized  users,  embedded  training  or  job  aids, 
and  quick  response  cross-referencing  capabilities 
with  related  manpower  documentation.  These 

capabilities  should  lead  to  increased  accuracy  and 
timeliness  of  requirements  production  and 
utilization  as  well  as  potential  economies  in 
administrative  support. 

(b)  Another  potential  aspect  of  this  component  is  the 
development  of  a  built-in  analytic  model  and 
mathematical  capability  to  increase  the  rigor  of 
manpower  requirements  determination.  For  example, 
whenever  mathematical  calculations  or  transforma¬ 
tions  are  required  the  formulas  should  be  readily 
accessed  on  the  system. 

It  should  be  noted  that  a  training  device  can  be  considered 
to  be  a  large  system  in  and  of  itself.  Thus,  it  has  its  own 
associated  resource  requirements.  Estimation  of  these 

additional  resource  requirements  is  beyond  the  current  scope 
of  ETES.  However,  it  is  expected  that  these  resource 
requirements  could  be  estimated  by  comparability  analysis. 
Input  to  the  determination  of  training  device  resource 
requirements  is  provided  by  the  task  requirements  associated 
with  each  device  which  are  determined  in  Procedure  2.0  and 
the  instructional  programs  developed  during  Procedure  4.5. 
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